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Eoreword 

I'he  Federal  Information  F’rocessing  Standards  Publication  Series  of  the  National  Bureau  of 
Standards  (NBS)  officially  publishes  standards  adopted  and  promulgated  under  the  provisions  of  Public 
Law  89-306  (Brooks  Act)  and  under  Part  6  of  Title  15,  Code  of  Federal  Regulations.  The  Brooks  Act  and 
subsequent  0MB  policy  guidance  designate  the  Department  of  Commerce  as  responsible  for  providing 
scientific  and  technological  advisory  services  for  the  management  of  Federal  ADP  procurement  and  usage, 
including  computer  networking.  These  legislative  and  executive  mandates  have  given  the  Department  of 
Commerce  important  responsibilities  for  improving  the  utilization  and  management  of  computers  and 
automatic  data  processing  systems  in  the  Federal  Government.  To  carry  out  these  responsibilities,  the 
NBS,  through  its  Institute  for  Computer  Sciences  and  Technology,  provides  leadership,  technical  guidance, 
and  coordination  of  Government  efforts  to  assist  in  the  development  of  guidelines  and  standards  in  these 
areas. 


James  H.  Burrows 

Director 

Institute  for  Computer  Sciences 
and  Technology 


Abstract 

These  guidelines  are  primarily  directed  to  people  who  operate  or  consume  remote  batch  computer 
service. 

J'he  evaluation  of  Remote  Batch  [Computer]  Service  (RBS)  is  dependent  on  several  factors,  including 
the  nature  of  the  service  provided,  the  RBS  equipment  and  its  staffing,  the  criteria  and  metrics  which  are 
deemed  applicable,  and  the  measurement  methodology.  These  guidelines  present  these  factors  in  a 
unifying  context  which  should  introduce  increased  orderliness  into  management  and  selection  of  RBS. 

Key  words:  Analysis;  computer;  evaluation;  Federal  Information  Processing  Standards  Publication; 
measurement;  performance;  remote  batch;  selection;  service. 
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Federal  Information  I'roeessing  Standards  [Publications  are  issued  by  the  National  Bureau  of  Standards  pursuant  to  tbe 
Federal  I’ropertv  and  Administrative  Services  Act  of  1949,  as  amended,  [’ublic  Law  89-306  (79  Stat.  1127),  Fxecutive 
Order  11717  (38  FR  12315,  dated  May  11,  1973)  and  Part  6  of  Title  15  Code  of  Federal  Regulations  (CFR). 

Name  of  Guideline:  (yuidelines  for  the  Measurement  of  Remote  Batch  Computer  Service. 

Category  of  Guideline:  ADP  Operations,  Computer  Performance  Management. 

Explanation:  These  guidelines  define  measures  and  describe  methodologies  for  measuring  remote  batch  computer 
network  service. 

Approving  Authority:  Department  of  Commerce,  National  Bureau  of  Standards  (Institute  for  Computer  Sciences  and 
Technology). 

Maintenance  Agency:  Department  of  Commerce,  National  Bureau  of  Standards  (Institute  for  Computer  Sciences  and 
Technology). 

Cross  Index:  See  bibliography. 

Applicability:  These  guidelines  are  a  basic  reference  document  to  inform  Federal  agencies  of  current  approaches  to 
evaluation  techniques  related  to  remote  batch  computer  service  (RBS).  These  guidelines  are  primarily  oriented  toward  the 
person  who  will  be  operating  a  computer  system  which  supports  RBS.  They  should  also  aid  both  consumers  and  providers  of 
RBS  in  the  specification  of  RBS  and  in  determining  whether  promised  services  have  been  delivered. 

Primary  Service  Attributes:  Availability,  reliability,  timeliness,  and  correctness  are  identified  as  the  primary  service 
attributes. 

Implementation:  These  guidelines  may  be  used  by  a  Federal  agency  which  has  a  need  to  measure  and  evaluate  the 
delivery  of  network  services.  The  methodology  and  procedures  described  provide  a  starting  point  for  such  network  related 
activities. 

Specifications:  Federal  Information  Processing  Standard  72  (FIPS  72),  Guidelines  for  the  Measurement  of  Remote  Batch 
Computer  Service,  (affixed). 

Qualifications:  The  recommendations  provided  in  these  guidelines  are  based  on  the  research  and  experience  of  not  only 
the  National  Bureau  of  Standards,  but  also  other  Federal  organizations  and  private  sector  groups.  In  the  future  these 
guidelines  will  be  modified  to  reflect  new  developments  in  network  performance  evaluation. 

These  guidelines  represent  recommended  good  practices  for  the  specification,  measurement,  evaluation,  and  selection  of 
remote  batch  computer  network  services.  These  guidelines  acknowledge,  but  do  not  address,  other  portions  of  the 
procurement  process  such  as  functional  demonstrations,  contractual  safeguards,  procurement  regulations  and  policy,  or  other 
ADP  procurement  considerations.  Thus,  in  order  to  be  consistent  with  overall  Federal  policy,  the  user  should  seek  current 
guidance  from  applicable  Office  of  Management  and  Budget  and  General  Services  Administration  policy  and  procurement 
directives. 

These  guidelines  are  not  procedural  steps  that  can  be  followed  as  a  “recipe”  with  successful  results.  Instead,  they  are 
intended  to  be  a  systematic  presentation  of  good  practices.  Nor  are  they  “clauses”  which  can  be  simply  inserted  into 
specifications  or  other  contractual  documents.  In  this  sense,  guidelines  are  useful  as  a  checklist  and  as  a  means  to  identify 
areas  where  special  competence,  expertise,  or  particular  attention  is  indicated. 
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These  guidelines  will  need  to  be  expanded  and/or  modified  as  further  knowledge  is  gained  of  the  techniques  involved. 
Comments,  critiques,  and  technical  contributions  directed  to  this  end  are  invited.  These  should  be  addressed  to  the  Associate 
Director  for  ADP  Standards,  Institute  for  Computer  Sciences  and  Technology,  National  Bureau  of  Standards,  Washington, 
D.C.  20234. 

Where  to  Obtain  Copies  of  the  Guideline:  Copies  of  this  publication  are  for  sale  by  the  National  Technical 
Information  Service,  U.S.  Department  of  Commerce,  Springfield,  Virginia  22161.  When  ordering,  refer  to  Federal 
Information  Processing  Standards  Publication  72  (FIPS-PUB-72)  and  title.  When  microfiche  is  desired,  this  should  be 
specified.  Payment  may  be  made  by  check,  money  order,  or  deposit  account. 
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Executive  Overview 

The  need  to  measure  remote  batch  computer  network  service  arises  in  the  selection,  operation,  evaluation,  and  testing  of 
such  services.  Availability,  reliability,  timeliness,  and  correctness  are  identified  as  the  primary  service  attributes  to  he 
measured  and/or  evaluated.  Other  Federal  documents  are  referenced  to  direct  the  reader  to  related  Federal  procurement  and 
other  ADF^  policies  and  procedures. 

These  guidelines  are  designed  for  use  by  Federal  officials  and  other  employees  who  have  responsibility  for  the  s})ecification, 
measurement,  operation,  evaluation,  or  selection  of  remote  batch  computer  service.  The  functional  performance  measures 
presented  in  this  document  can  he  used  to  evaluate  the  delivery  of  remote  batch  service  from  Government-operated  comjmter 
systems. 

I'he  current  state-of-the-art  in  the  evaluation  of  computer  service  makes  it  necessary  to  j)erform  comparative  rather  than 
absolute  evaluation.  It  is  always  necessary  to  qualify  measurements  by  a  description  of  the  measurement  environment  and 
methodology.  Measurement  may  he  done  under  controlled  conditions,  implying  a  test  environment  where  it  is  desirable,  if 
not  absolutely  necessary,  that  tests  be  repeatable. 

The  selection  of  measurement  methodology  must  incorporate  consideration  of  the  conditions  under  which  measurement  is 
conducted  and  the  functional  performance  measures  by  which  the  interactive  computer  service  is  to  he  judged.  Cost, 
complexity,  and  accuracy  must  weigh  in  the  selection.  The  methodologies  available  include  measurement  autornaticallv  by 
the  central  host  computer,  the  remote  batch  device,  and  manually  employing  printing  time  clocks.  The  statistical  methods 
used  for  analyzing  the  data  and  generating  reports  must  be  carefully  designed  and  tested. 

Throughout  these  guidelines,  specific  summary  guidance  is  set  in  boldface  italic  type  so  that  it  is  easily  identified  and 
located.  This  is  boldface  italic  type. 

A  glossary  provides  definitions  and  discussion  of  terms  and  a  bibliography  is  provided. 
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Summary  Guidance 


Objectives  of  this  Guideline:  These  guidelines  should  aid  both  consumers  and  providers  of  remote  hatch  computer 
services  (RBS)  in  determining  whether  promised  services  have  been  delivered.  Availability',  reliability',  timeliness,  and 
correctness  are  identified  as  the  primary  service  attributes  to  be  measured  and  evaluated. 

Applicability  of  this  Guideline:  The  definitions  of  remote  batch  service  quality  should  be  agreed  u[)on  and  [)uhlished 
in  advance  of  any  attempt  to  measure  that  service.  These  guidelines  will  assist  the  user  of  remote  hatch  services  in 
establishing  formal  metrics  which  define  remote  batch  service  quality. 

Where  service  levels  are  important,  they  should  be  contractually  agreed  upon  and  stated  in  a  manner  so  that  they  can  be 
monitored. 

Specifying  Metrics  W'hich  Define  Remote  Batch  Service  Quality:  A  remote  batch  service  should  be  measured  on 
criteria  involving  functional  and  performance  characteristics.  The  performance  characteristics  should  be  evaluated  in  terms  of 
metrics  that  are  meaningful  to  the  consumer  and  that  are  measurable.  Consumers’  interests  are  in  whether  the  remote  batch 
service  is  available  when  they  need  it,  whether  it  operates  without  interruption,  whether  it  provides  their  results  when  they 
need  them,  and  whether  the  outputs  are  correct.  These  attributes  are  termed  availability,  reliability,  timeliness,  and 
correctness.  The  evaluation  of  a  remote  batch  service  must  be  on  the  basis  of  its  handling  of  individual  submissions. 

Availability:  Availability  can  be  specified  in  terms  of  the  portion  of  a  selected  time  interval  during  which  the  remote  site 
is  capable  of  performing  its  functions.  Availability  must  be  specified  for  each  remote  site  individually.  If  a  metric  is  required 
for  the  RBS  as  a  whole,  the  appropriate  one  is  the  portion  of  all  specified  remote  sites  that  meet  the  requirements  for 
availability. 

Reliability:  The  reliability  of  an  RBS  involves  the  ability  of  the  remote  batch  device  to  operate  without  the  consumer’s 
corrective  intervention  through  a  complete  input  or  output  operation.  When  each  of  the  operations  performed  by  a  remote 
terminal  device  has  a  different  set  of  hardware  that  can  impact  reliability,  the  required  level  of  reliability  can  be  specified 
for  each. 

Timeliness:  Timeliness  is  usually  equated  to  “turnaround,”  the  elapsed  time  between  the  moment  of  submission  and  the 
instant  that  output  is  returned.  Timeliness  objectives  should  be  stated  as  the  portion  of  instances  in  which  specified  timing 
objectives  are  met.  When  two  or  more  output  devices  are  employed,  timeliness  metrics  must  identify  which  devices  are 
applicable  to  each.  Specification  or  measurement  of  a  remote  batch  system’s  timeliness  requires  the  identification  of  events 
signifying  the  submission  time  and  the  completion  time. 

Three  main  RBS  inputs  exist:  Provider  Operated,  Consumer  Operated,  and  Interactive  Terminal.  Output  from  the  remote 
batch  service  may  be  Immediately  Transmitted  to  the  remote  batch  device,  it  may  be  Held  Until  Requested  by  the  consumer, 
or  it  may  be  Delivered  to  the  consumer  by  a  courier. 

Given  the  various  input,  processing,  and  output  alternatives,  the  submission  and  completion  phases  must  be  carefully 
defined  for  metrics  to  be  clear.  For  immediate  output  the  completion  time  is  the  instant  when  the  last  record  of  output  is 
printed.  For  request  output  the  completion  time  is  the  instant  when  the  user  is  notified  of  output  availability,  or  when  the 
last  record  of  output  is  available  for  transmission  upon  request.  When  the  provider  employs  a  courier  service  for  voluminous 
or  special  output,  appropriate  metrics  should  be  employed  to  measure  the  effect  on  the  delivery  of  output.  For  database 
updates  the  completion  time  is  the  instant,  upon  completion  of  the  database  update,  when  the  revised  data  can  be  accessed. 

Correctness:  The  current  state-of-the-art  for  measuring  correctness  largely  depends  on  rerun  rates  for  determining 
correctness  levels.  Agreement  is  needed  on  the  mechanism  for  determining  that  a  job  must  be  rerun  because  of  the 
provider’s  actions.  Required  levels  of  service  should  be  specified  in  terms  of  the  portion  of  all  submissions  that  may  be 
affected  by  these  conditions. 

Selecting  the  Measurement  Methodology:  The  decision  to  measure  RBS  must  incorporate  a  selection  of  appropriate 
measurement  methodology.  The  need  to  perform  statistical  analysis  on  the  measured  data  strongly  favors  a  measurement 
methodology  which  provides  its  data  in  machine  processible  form.  Data  acquisition  and  analysis  must  be  specified  in 
advance.  The  statistical  criteria  and  sample  size  should  be  chosen  in  advance  of  any  measurement  of  service  quality. 
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In  most  situations,  evaluation  will  be  made  under  partially  controlled  conditions  where  the  analyst  enters  submissions  in 
addition  to  the  normal  workload  on  the  remote  batch  service.  Measurement  under  partially  controlled  conditions  may  require 
measurement  under  uncontrolled  conditions  in  order  to  provide  comparison  values  for  detection  of  uncontrolled  changes. 
Measurement  under  total  control  is  available  to  the  provider  of  the  RBS.  It  should  be  pursued  only  if  partial  control  is 
inadequate  and  only  after  thorough  design  of  the  experiment. 

Performing  the  Remote  Batch  Service  Measurement:  Even  though  measurement  technology  could  stand  some 
improvement,  there  is  presently  a  sufficient  basis  for  remote  batch  service  measurement.  The  recommended  approach 
consists  of  the  four  steps  outlined  below. 

1.  Identification  of  provider  and  consumer:  The  first  step  in  RBS  measurement  is  the  identification  of  the  provider  and 
consumer.  This  is  not  necessarily  trivial  when  dealing  with  groups  of  consumers  who  may  belong  to  different  organizational 
units  and  who  may  deal  with  the  provider  through  different  spokespersons. 

2.  Selection  of  descriptive  scenario:  The  second  step  is  selection  of  the  scenario  descriptive  of  how  the  consumer  obtains 
RBS.  The  scenarios  presented  in  this  report  should  serve  as  a  starting  place.  Both  provider  and  consumer  should  agree  on 
the  representativeness  of  the  scenario. 

3.  Selection  of  Metrics:  The  third  step  is  the  selection  of  RBS  metrics.  This  selection  must  be  done  in  concert  with  the 
fourth  step,  selection  of  measurement  methodology.  The  problem  is  to  determine  what  should  be  measured  and  what  can  be 
measured.  The  consumer  group  should  be  kept  aware  of  and  educated  about  the  selections  of  metrics  and  methodology  and 
the  logic  which  led  to  that  selection. 

4.  Implementation:  The  final  step  is  implementation.  RBS  performance  evaluation  may  be  employed  to  settle  disputes,  to 
contractually  establish  definitions  of  satisfactory  RBS,  to  support  negotiation  or  litigation  when  agreed-upon  service  criteria 
are  violated,  to  alert  provider  management  when  service  levels  are  out-of-bounds,  or  to  help  consumers  select  a  new  RBS. 


8 


GUIDELINES  FOR  THE  MEASUREMENT  OF 
REMOTE  BATCH  COMPUTER  SERVICE 


KII’S  I’l  li  72 


1.  Introduction 

These  guidelines  should  aid  both  consumers  and 
providers  of  remote  batch  computer  services  in 
determining  whether  promised  services  have  been 
delivered.  Availability,  reliability,  timeliness,  and 
correctness  are  identified  as  the  primary  service  attributes 
to  be  measured  and  evaluated.  There  are  specific 
recommendations  distributed  throughout  the  text.  These 
recommendations  are  set  in  boldface  italic  type. 

1.1  How  to  Read  and  Use  These  Guidelines 

The  major  sections  of  this  document  discuss 
measurement  of  remote  batch  service  in  terms  relevant  to 
providers  and  consumers.  The  phases  of  input,  processing, 
and  output  are  defined  in  context.  Metrics  are  defined  and 
meaurement  methodologv  developed,  with  consideration  of 
the  measurement  environment. 

An  overview  of  each  section  is  followed  by  specific 
recommendations  and  discussions  of  salient  points. 
Throughout  the  document,  specific  summary  guidance  is 
set  in  boldface  italic  type.  This  document  is  organized  into 
five  major  sections  each  of  which  contains  subsections 
which  address  specific  topics  in  increasing  depth.  The 
reader  is  advised  to  read  the  document  in  its  entirety 
before  attempting  to  apply  the  guidance  for  a  particular 
purpose. 

1.2  Applicability 

Application  of  the  metrics  suggested  in  this  document 
can  usually  be  applied  to  lend  rigor  to  the  analyst’s 
“intuitive  feel”  that  is  so  often  incorrect.  Formal 
metrics  should  be  applied  to  situations  ivhere  the 
issue  is: 

Setting  contract  terms. 

Selection  of  one  remote  batch  service  from 
among  many. 

Performance  improvement  or  performance 
projection, 

Performance  management  system  design. 

The  application  of  these  metrics  is  not  an  adequate 
solution  to  all  the  problems  that  are  tvpically  encountered 
in  dealing  with  these  issues,  of  course.  In  each  situation  a 
mix  of  skills  is  required:  for  example,  setting  contract 
terms  in  a  formal  contract  requires  the  skills  of  a 
contracting  officer  or  attorney.  Properly  applying  the 
metrics,  however,  can  improve  the  results  in  these 
situations  when  a  remote  batch  service  is  of  concern. 


1.2.1  Setting  Contract  Terms 

A  contract  for  remote  hatch  service  from  a  provider 
typically  sets  the  charging  rate  for  usage,  but  does  not 
provide  incentives  for  exceptionallv  good  service  or  set 
minimum  levels  of  service.  Where  service  levels  are 
important,  they  should  be  contractually  agreed 
upon,  and  stated  in  a  manner  so  that  they  can  be 
monitored. 

1.2.2  Remote  Batch  Service  Selection 

A  remote  batch  service  should  be  selected  on 
criteria  involving  both  functional  and 
performance  characteristics.  The  performance 
characteristics  should  be  evaluated  in  terms  of 
metrics  that  are  meaningful  to  the  consumer  and 
that  are  measurable.  A  good  starting  place  is  the 
metrics  given  in  this  report. 

The  evaluation  of  competitive  remote  batch 
services  should  be  based  on  satisfactory- 
performance  unless  added  utility  is  associated 
with  performance  above  this  minimum  level.  In 
most  situations,  evaluation  will  be  made  under  partially 
controlled  conditions  where  the  analyst  enters  submissions 
in  addition  to  the  normal  workload  on  the  remote  hatch 
service. 

1.2.3  Performance  Improvement  and 

Performance  Projection 

Determining  ways  and  means  to  improve  performance 
is  an  exercise  with  much  similarity  to  forecasting  system 
performance.  In  both  cases  analysts  must  hypothesize 
relationships  and  then  test  these  hypotheses.  In  both  cases 
measurement  under  no  control,  partial  control,  or  total 
control  may  be  required. 

The  results  of  performance  improvement  efforts  should 
be  evaluated  in  terms  of  their  utility  to  consumers — 
employing  metrics  that  are  meaningful  in  those  terms. 
Similarly,  the  objective  of  a  performance  projection  study 
should  be  to  establish  values  of  metrics  that  indicate 
service  levels  to  consumers. 

1.2.4  Performance  Management  System 

A  performance  management  system  |6]  enqiloys 
measurement  to  compare  realized  values  against 
previously-established  values.  The  previously-established 
values  may  involve  utilization  of  system  resources,  the 
level  of  loading,  and  the  achieved  levels  of  service  from 
the  computer  system.  In  the  case  of  remote  batch  services. 
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the  level  of  service  is  in  terms  of  the  metrics  described  in 
this  document. 

A  performance  management  system,  like  most  other 
applications  of  metrics  for  describing  service  levels  on 
remote  batch  services,  requires  explicit  efforts  to  choose 
both  the  appropriate  metrics  and  their  values.  In  fact,  this 
is  probably  the  most  difficult  part  of  designing  a 
performance  management  system;  an  organized  approach  is 
needed  to  ensure  that  comparisons  are  meaningful  and  do 
not  lead  to  unjustified  attempts  to  change  the  RBS. 


1.2.5  Setting  Service  Level  Metrics  and 
Objectives 

Setting  timeliness  objectives  is  a  challenging  [)rocess 
due  to  the  variety  of  possibilities  (including  submission 
events,  completion  events,  turnaround  objective,  deadline, 
and  level  of  achievement).  Generally  the  most  appropriate 
approach  is  to  set  from  two  to  ten  turnaround  time 
objectives  with  a  different  deadline  for  each  submission  for 
which  such  an  objective  is  appropriate.  Since  some 
submissions  will  not  meet  the  objectives,  the  level  of 
achieving  each  objective  can  be  measured  and  compared 
with  nominal  values.  Alternatively,  the  number  of 
submissions  meeting  and  failing  different  objectives  can  be 
combined  when  this  technique  matches  the  needs  of  the 
remote  batch  service’s  provider  and  consumers. 

The  steps  to  set  the  objectives  for  a  particular  regular 
submission  are  as  follows: 

1.  Pick  the  events  indicating  submission  and  completion 

times.  The  submission  event  might  be,  for  example, 
the  instant  the  first  record  is  read  through  a  card 
reader  that  is  part  of  a  remote  batch  device,  and  the 
completion  event  might  be  the  last  line  of  printing 
produced  for  the  submission. 

2.  Choose  whether  the  timeliness  objective  should 

involve  deadline  or  turnaround  time. 

.3.  Determine  the  value  for  turnaround  time,  or  the  times 
for  submission  and  completion  deadlines. 

4.  Select  the  portion  of  all  such  submissions  that  must 
meet  the  timeliness  criterion  (e.g.,  90%). 

The  improved  rigor  of  well-defined,  specifically- 
designed  metrics  for  a  remote  batch  service  can  reduce 
confusion  in  performance  analyses  and  reduce  disputes 
about  achieved  service  levels  in  operational  systems. 
Choosing  metrics  and  selecting  their  nominal  values  is 
feasible  and  rewarding.  If  the  service  level  is  worth 
arguing  or  worrying  about,  it  is  worth  defining  and 
measuring. 


1.3  The  Relationship  Between  Providers  and 

Consumers 

Although  there  is  much  advice  available  on  procuring 
remote  batch  service  (RBS)*  (c.f.  [2,  3,  4]),  consumers  of 
an  RBS  have  few  indicators  of  poor  service  other  than  late 
outputs.  The  common  metrics  of  service  (e.g.,  turnaround 
time  within  the  computer  averaged  over  all  submissions) 
do  not  indicate  how  well  the  remote  batch  service  is 
actually  serving  the  consumers.  Lack  of  indicators  also 
hampers  RBS  providers  from  assessing  and  improving  the 
quality  of  the  service  provided.  This  objective  data  about 
RBS  can  help  bridge  the  gap  between  consumers  and 
providers  of  remote  batch  services  and  provide  a  common 
ground  for  communication  between  the  two  groups. 

An  organization’s  ability  to  carry  out  its  operations 
under  budgetary  constraints  is  often  dependent  on  the 
results  from  the  remote  batch  service.  Many  organizations 
have  successfully  improved  communication  between 
provider  and  consumer  by  setting  explicit  service  level 
objectives. 

1.4  Special  Problems  of  Remote  Batch  Service 

Metrics 

Measurement  of  RBS  is  complex  because  of  the 
variability  of  the  communications  facility,  and  the  limited 
input  and  output  capabilities  available  to  serve  individual 
consumers.  Each  of  these  factors  has  to  be  considered  in 
developing  metrics  appropriate  for  describing  the  remote 
batch  service.  The  effect  of  these  factors  in  developing 
specific  metrics  for  all  phases  of  RBS  is  explained  in 
section  2. 

For  example,  consumer  actions  affect  measurement  of 
device  availability.  In  many  remote  batch  services  the 
consumers  operate  the  remote  batch  device  to  input  their 
submissions.  In  addition,  they  may  operate  the  device  to 
allow  the  return  of  their  output.  On  the  input  side, 
consumers  become  immediately  aware  if  the  device  refuses 
to  accept  their  submissions.  This  may  occur  if  the 
communication  facility  cannot  accept  the  submission  and 
no  capability  exists  for  the  device  to  store  the  submission 
for  later  transmission.  This  situation  can  also  occur  when 
the  device  itself  is  inoperable  and  maintenance  is  required 
at  the  remote  site.  Metrics  describing  availability  of  the 
service  must  recognize  that  the  consumers  with 
submissions  will  sometimes  remain  at  the  device  to  be  able 
to  employ  it  as  soon  as  possible  after  failure  to  submit 
input. 

Output  from  the  RBS  may  also  be  dependent  on 
consumers’  actions.  As  an  example,  the  consumer’s  failure 
to  load  paper  in  a  printer  at  the  remote  site  will  delay 
delivery  of  the  output  through  no  fault  of  the  provider.  In 


‘Terms  specific  to  the  subject  of  this  publication  are  defined  in  the 
glossary. 
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addition,  some  services  hold  output  until  it  is  requested  by 
the  consumers.  If  the  consumers  do  not  take  action  to 
request  the  output,  then  the  recorded  turnaround  time  of 
the  submission  will  be  very  high  in  spite  of  the  providers’ 
exemplary  services.  Therefore,  metrics  describing 
timeliness  of  the  service  must  reflect  the 
potential  impact  of  the  consumers. 

Similarly,  the  cost  and  reliability  of  the  communication 
facility  affect  service  delivery.  Since  these  facilities  have 
specified  maximum  transmission  capabilities,  the  return  of 
voluminous  output  can  require  a  considerable  time.  If 
consumers  have  chosen  a  particular  line  capacity  or 
transmission  rate  and  have  large  volumes  of  output,  the 
most  appropriate  measure  of  turnaround  time  may  be  one 
that  does  not  include  the  output  transmission  time. 

Even  with  a  high-speed  communication  line,  the  input 
or  output  capacity  of  a  remote  batch  device  may  cause  the 
time  for  these  operations  to  dominate  the  total  time  for 
responding  to  a  submission.  In  addition,  the  delay 
experienced  in  printing  one  consumer’s  output  may  affect 
other  consumers  dramatically.  Excessive  output  from  one 
consumer  can  cause  subsequent  consumers  to  wait  for  long 
periods  before  their  output  even  begins  to  be  printed. 
Output  production  in  a  conventional  batch  system  is  less  of 
a  problem  due  to  the  economic  viability  of  high  capacity 
printers  and  computer  output  to  microfilm  (COM) 
equipment.  When  the  demand  for  output  production  is 
spread  among  a  number  of  remote  sites,  these  high  speed 
printers  are  not  economically  viable:  as  a  result,  a  few  very 
large  jobs  can  have  an  adverse  impact  on  consumers’ 
service  levels. 

2.  Remote  Batch  Scenarios 

The  first  step  in  developing  effective  measures  of  RBS 
is  identification  of  the  phases  of  RBS  and  the  tasks 
involved  in  each  phase.  A  consumer  may  employ  a  remote 
batch  service  several  times  in  the  course  of  solving  a 
problem  or  editing  some  data.  The  consumer’s  overall 
objective  is  to  complete  the  work  task,  not  to  obtain 
output.  Specifying  each  work  task  and  evaluating  a  remote 
batch  service  by  task  completion  time  would  be  ineffective 
due  to  the  dependence  on  highly  variable  human  problem¬ 
solving  capabilities,  the  inordinate  effort  required  to 
specify  the  task  and  its  completion  time,  and  the  difficulty 
in  separating  human  from  computer  effects.  Therefore, 
the  evaluation  of  a  remote  batch  service  must  be 
on  the  basis  of  its  handling  of  individual 
submissions. 

Scenarios  of  RBS  usage  begin  with  some  specific  act  of 
submission  and  end  with  (or  before)  the  return  of  all 
output. 

A  scenario  for  using  a  remote  batch  service  can  be 
divided  into  three  general  phases — input,  processing,  and 
output.  The  specifics  of  each  of  these  phases  may  indicate 


widely  different  applications  of  the  RBS,  and  sugge.st  that 
different  metrics  are  applicable.  Since  different  activities 
and  queues  exist  in  various  systems,  the  consumer 
perceives  different  system  characteristics.  However, 
common  features  of  the  three  phases  can  be  used  to 
facilitate  comparisons  of  services  though  these  phases  may 
not  be  applicable  to  all  systems. 

For  example,  the  functional  phases  below  may  not 
describe  a  specific  RBS  exactly  because  some  new  products 
can  function  as  remote  batch  devices,  and  as  interactive 
terminals  and  stand-alone  minicomputers  as  well.  I'he 
general  RBS  phases  are  blurred  as  a  result  of  these 
enhanced  capabilities. 

In  addition,  different  orientations  (both  applications 
and  degree  of  service)  may  apply  to  remote  batch  services. 
Therefore,  the  description  of  each  of  the  general  phases 
begins  with  discussion  of  the  more  important  differences 
that  may  influence  the  mapping  from  actual  RBS  activities 
to  the  functional  phases. 

2.1  Input 

Input  to  a  remote  batch  service  can  be  a  relatively 
simple  action,  such  as  a  consumer  placing  a  stack  of 
punched  cards  into  the  hopper  of  a  remote  batch  device 
and  pressing  a  button.  Input  may  also  consist  of  several 
sequential  steps,  with  the  consumer  only  executing  a  few 
of  them.  Three  main  remote  batch  service  input 
environments  exist,  differing  in  the  degree  to  which  the 
provider  takes  responsibility  for  elements  of  input 
operations.  These  are  identified  below  as  Provider 
Operated,  Consumer  Operated,  and  Interactive  Terminal. 

2.1.1  Provider  Operated  Input 

In  this  environment,  the  provider  of  the  remote  batch 
service  furnishes  the  remote  batch  devices  and  maintains 
personnel  to  operate  them.  The  modes  of  input  are  as 
follows:  the  consumer  may  physically  place  the  input 
submission  media  into  a  designated  area;  the  consumer 
may  notify  the  provider-supplied  personnel  that  data  have 
been  input  through  a  data-entry  device;  or  the  consumer 
may  actually  hand  the  submission  to  the  provider-supplied 
personnel  at  the  remote  site.  The  actual  time  of  input  is 
when  the  consumer  takes  the  necessary  actions  to  signify 
transfer  of  control.  While  the  consumer  will  perceive  that 
this  type  of  service  is  virtually  always  available,  the 
provider  does  not  have  an  automatic  means  of  determining 
the  time  of  input. 

In  the  absence  of  automatic  means  to  record  the  time  of 
input,  the  RBS  provider  may  need  to  maintain  appropriate 
manual  records  to  record  input  time.  Since  the  consumer 
has  completed  all  the  actions  to  initiate  the  input  process, 
and  control  of  the  submission  has  been  effectively 
transferred  to  the  provider,  successful  input  to  the  remote 
batch  service  is  completed.  Provider-supplied  personnel 
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have  control  of  the  consumer’s  submission  media  and  must 
now  perform  additional  intermediate  actions  to  transmit 
the  submission  to  the  central  site  and  to  ensure  its 
acceptance  there  for  processing.  These  actions  may  include 
some  delays  while  equipment  and/or  personnel  are 
involved  in  activities  for  other  submissions. 


2.1.2  Consumer  Operated  Input 

In  this  environment,  control  of  the  submission  is  not 
transferred  to  the  provider  until  the  consumer’s  attempt  to 
do  so  is  successful.  This  attempt  may  consist  of  pressing 
the  START  button  on  a  remote  card  input  device, 
initiating  a  communication  line  after  loading  a  floppy  disk, 
or  entering  a  command  to  initiate  the  transfer  of  records. 
The  submission  time  is  the  instant  when  the  first  record  is 
read. 

The  equipment  at  the  consumer-operated  remote  site 
might  be  provided  by  the  same  organizational  unit  which 
operates  the  central  computer.  Alternately,  it  might  be 
provided  by  the  consumer’s  organization,  or  even  by  a 
third  party. 


2.1.3  Interactive  Terminal  Originated  Input 

An  interactive  device  can  be  used  to  input  records  to 
storage  media  at  a  central  site,  and  subsequently  to 
command  that  those  records  be  interpreted  as  a  batch 
submission.  Remote  batch  systems  usually  include  this 
capability  when  the  computer  at  the  central  site  supports 
an  interactive  service  as  well  as  the  RBS. 

The  submission  time  for  interactive  terminal  origination 
is  the  time  of  successful  request  for  batch  submission,  i.e., 
when  the  RBS  command  transaction  is  accepted  by  the 
RBS.  Submission  may  be  indicated  by  the  completion  of 
printing  on  a  printer,  the  availability  of  results  on  a 
terminal,  or  some  other  possibility  that  an  inventive 
provider  creates. 


2,1.4  Input  Functional  Phases 

A  remote  batch  submission  may  go  through  several 
functional  phases,  each  of  which  introduces  the  possibility 
of  a  delay  or  an  error.  Figure  1  represents  these  phases; 

11 —  Queue  for  Provider-Supplied  Personnel  Pickup 

12 —  Queue  for  Input  Device 

13 —  Input  of  Physical  Media  or  Command 

14 —  Queue  for  Transmission 

15 —  Transmission 


The  phases  that  may  exist  for  the  three  classes  of  input 


situations  are: 

Interactive  Terminal: 

15 

(Transmission) 

Consumer  Operated; 

12 

(Queue  for  Input) 

13 

(Input  of  Physical  Media) 

14 

(Queue  for  Transmission) 

15 

(Transmission) 

Provider  Operated: 

11 

(Queue  for  Pickup) 

12 

(Queue  for  Input) 

13 

(Input  of  Physical  Media) 

14 

(Queue  for  Transmission) 

15 

(Transmission) 

Each  of  the  three  input  environments  includes  the  three 
functional  phases,  but  the  implementations  of  these 
functional  phases  may  be  quite  different  in  each  actual 
system. 

The  primary  importance  of  identifying  these  phases  is 
to  facilitate  measuring  the  timeliness  of  the  remote  batch 
system.  Accordingly,  the  descriptions  below  emphasize  the 
elapsed  time  between  transition  of  the  phases. 


Queue  for  Pickup — If  a  clerk  is  busy  with  other  activities, 
a  submission  may  await  pickup  for  a  long  time.  For 
this  reason  many  remote  sites  maintain  sign-in  logs 
or  they  time  stamp  job  submission  cards  so  that  the 
consumer  can  record  the  time  of  submission. 

Queue  for  Input — After  pickup,  a  submission  may  remain 
outside  the  automated  system  waiting  for  the  input 
device  to  be  available.  Usually,  the  duration  of  the 
wait  in  this  queue  is  quite  small.  If  this  queue  delay 
is  not  small,  it  may  be  an  indication  that  the  remote 
batch  device  and/or  the  transmission  rate  is 
inadequate.  When  the  device  is  provider  operated, 
the  attendant  should  note  this  queuing  delay  and 
report  it  to  the  appropriate  manager.  When  the 
device  is  consumer  operated,  the  consumers  need  to 
have  established  norms  for  queuing  delay  and  need 
to  know  to  whom  to  report  excessive  delays. 

Input  of  Media — The  most  obvious  media  for  input  are 
punched  cards.  However,  advancing  technology  has 
led  to  rather  widespread  use  of  more  sophisticated 
media  such  as  floppy  disks.  In  some  remote  sites  a 
kev-to-disk  system  is  employed  to  enter  data  for 
remote  batch  services.  In  other  cases  the  data  is 
entered  as  a  disk  which  is  then  transferred  to  the 
remote  batch  device. 
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ueue  for  TransmisS'ion — The  communication  line  mav  be 
inoperable  or  already  in  use  at  the  instant  the  new 
submission  arrives.  In  this  case,  the  submission  will 
await  the  availability  of  the  line.  Queuing  for 
transmission  usually  occurs  when  the 
communication  capability  is  temporarily  dedicated 
to  send  output  back  to  the  remote  site. 


fransmission — If  the  communication  line  is  of  relatively 
low  bandwidth  and  the  submission  includes 
relatively  large  quantities  of  data,  the  time  for 
transmission  of  the  submission  will  be  large.  If 
consumers  are  operating  the  remote  batch  device, 
they  may  be  particularly  sensitive  to  transmission 
time  if  they  are  reijuired  to  remain  at  the  remote 
site  to  ensure  correct  transmission  of  their 
submissions. 


2.2  Processing 

The  input  phase  begins  when  the  provider  gains  control 
over  the  consumers’  submission.  The  })rocessing  phase 
begins  when  the  submission  is  available  at  the  central  site. 


2.2.1  Initiation  Delays 


Since  most  data  processing  installations  cannot  process 
jobs  instantaneously  upon  arrival,  the  actual  processing  of 
the  consumers’  submissions  is  tvpicallv  delayed  in  a 


central  site  initiation  queue.  The  consumers  should  be 
unconcerned  about  the  mechanisms  the 
providers  have  established  for  classifying  and 
processing  submissions,  tis  long  as  the  system 
operates  as  expected.  However,  the  providers 
must  consider  both  the  mechanisms  and  the 
effects  in  order  to  meet  their  commitments  to  the 
consumers. 


2.2.2  Processing  Objeetives 

The  objecti\e  of  processing  mav  be  to  obtain  the  output 
from  e.xecuting  a  job,  or  it  may  be  to  u|)date  a  database.  In 
tlie  first  case  the  completion  time  of  the  remote  batch 
submission  is  when  the  out})Ut  is  returned.  In  the  rase  of 
database  update  the  com|detion  time  mav  be  the  instant 
that  the  database  has  been  updated  and  the  results  are 
available  for  retrieval. 

Although  the  objective  of  a  database  update  submission 
may  be  fulfilled  at  the  instant  the  results  are  available  for 
inquiry,  a  second  objective  mav  be  equally  impcirtant:  to 
notify  the  consumer  of  the  completion  of  processing  so  that 
further  processing  can  be  initiated.  In  cases  tvhere  the 
objective  of  a  submission  is  database  update,  two 
measures  of  timeliness  may  be  required  to  reflect 
both  the  completion  of  the  update  and  the 
notification  of  the  update's  completion. 
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2.2.3  Functional  Phases 

The  fLinctional  ste[)s  in  the  processing  phase  of  a 
remote  hatch  system  are  not  as  a[)parent  to  the  consumer 
as  the  input  steps.  Figure  2  portrays  these  phases: 

PI — Queue  for  Processing 
P2 — Initial  Processing 
P3 — Database  Li|)(Iate 
P4 — Final  Processing 


Figure  2.  Processing  phases 


T 


P2 


P3 


i 


These  phases  are  more  general  than  usual  descriptions 
of  [irocessing  steps.  Details  of  processing  steps  are  of  little 
interest  to  the  remote  hatch  consumer  other  than  for  their 
effects  on  the  completion  of  the  requested  task. 


Database  Update — For  submissions  re(]uiring  a  database 
update,  this  phase  begins  with  the  first  modification 
to  the  database;  it  ends  when  the  last  modification 
is  completed  and  the  database  is  available  for 
accessing.  The  beginning  and  ending  times  for  this 
phase  are  not  always  recorded  automatically,  so 
application  programs  may  be  required  to  collect  the 
timing  information. 

Final  Processing — This  phase  occurs  onlv  for  submissions 
that  include  a  database  update.  It  includes  all 
processing  that  takes  [)lace  between  the  completion 
of  the  update  and  time  when  all  output  is  available. 

2.3  Output 

Consumers  usually  regard  their  out[)ut  as  the  objective 
when  making  a  remote  batch  submission.  (Only  in  a  few 
cases  such  as  database  update  is  no  output  produced.)  Most 
remote  batch  devices  either  have  a  printer  associated  with 
them  or  they  may  share  a  printer  with  other  devices  (e.g., 
interactive  terminals  used  as  remote  batch  devices).  In 
some  cases  output  may  be  prepared  at  the  central  site  and 
delivered  to  remote  sites. 

2.3.1  Delivered  Output 

I'he  primary  advantage  of  supplementary  delivered 
output  service  is  that  large  volumes  of  output  can  be 
produced  without  the  expense  of  high  speed 

communication  lines  and  multiple,  high  speed  remote 
printers.  Special  media  conversion  may  also  be  required 
for  the  consumer’s  output;  central  conversion  is  tvpicallv 
the  only  economically  feasible  technique.  For  example, 
multiple  copies  may  be  re<]uired  or  microfiche  may  be 
needed.  In  such  cases  the  output  is  processed  centrally  and 
the  resultant  media  are  transported  to  the  demote  site. 

ff  hen  the  provider  employs  a  courier  sert'ice 
for  voluminous  or  special  output,  appropriate 
metrics  should  be  employed  to  measure  the  effect 
on  the  delivery  of  output. 


Queue  for  Processing — This  phase  begins  at  the  instant 
the  entire  submission  arrives  at  the  central  site. 
Submissions  may  be  sorteif  so  that  they  will  begin 
processing  in  an  order  different  from  their  arrival 
order.  Both  the  beginning  and  end  of  this  phase  can 
usually  be  determined  automatically  by  the  provider 
through  the  operating  system  or  th<*  communica¬ 
tions  monitor. 

Initial  Processing — This  [ihase  includes  all  processing 
except  database  updates.  Although  conipletion  of 
this  phase  may  be  difficult  to  determine  when  a 
database  update  phase  follows,  its  completion  time 
is  automatically  recorded  in  most  operating  systems 
when  It  is  the  final  processing  phase. 


2.3.2  Immediate  vs  Request  Output 

The  difference  between  immediate  and  request  output 
affects  measures  of  timeliness  (discussed  in  3.3  below). 
Production  of  immediate  output  usually  employs  a  first-in- 
first-out  scheduling  discipline.  The  output  is  sent  to  the 
remote  site  as  soon  as  it  is  available,  the  communication 
line  is  free,  and  the  remote  batch  device  is  able  to  operate. 

Very  often  request  output  techniques  are  more  effective 
for  the  consumer.  This  means  that  the  output  is  held  on 
storage  media  where  it  is  available  for  the  consumer  as 
soon  as  processing  is  completed.  If  the  consumer  does  not 
request  the  output,  output  from  other  consumers  can  be 
[)rinted.  In  addition,  the  consumer  may  be  able  to  review 
the  output  with  an  interactive  terminal  prior  to  having  it 
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printed  and  then  (iecide  whether  it  ^h()uld  he  ininie<liatelv 
discarded  rather  than  printed.  Although  the  output 
ca[)ahility  of  the  remote  hatch  service  is  used  less 
efficiently  and  is  idle  more  freipientiv,  the  service  level 
provided  by  the  remote  batch  service  may  he  higher  due  to 
the  reduction  in  total  outfuit  printed. 

Timeliness  metries  for  immediate  output 
usually  begin  with  the  submission,  and  end  when 
all  the  output  has  been  printed.  F.tnploying  these 
metrics  for  reijuest  output  can  be  deceptive  because  the 
output  mav  never  be  [)rinted.  .4ssignnient  of  arbitrary 
eompletion  times  for  otitput  never  printed 
provides  little  information  and  may,  in  faet, 
interfere  with  timeliness  measurements.  The 
proper  completion  time  for  a  submission 
employing  request  output  is  the  instant  that 
output  is  available  for  its  printing  to  begin. 

2.3.3  Functional  Phases 

The  functional  phases  in  output  may  be  executed 
several  times  for  a  single  submission  in  order  to  provide 
several  modes  of  output.  For  example,  a  consumer  might 
have  some  output  immediately  transmitted  to  the  remote 
site,  with  the  bulk  of  the  output  physically  delivered  to  the 
site.  On  the  other  hand,  some  output  may  he  produced  by 
only  a  subset  of  the  phases  shown  in  Figure  3. 

01 — Queue  tor  Transmission 

02 — Transmission 

03 — Queue  for  Output  Process 

04 — Output  Process 

05 — Queue  for  Courier 

06 — Courier  Transfer 

07 — Output  Return 


The  detailed  phases  OI  through  04  occur  withiti  the 
automated  processing  system  of  the  com|)ut(‘r  and  tin* 
communication  systems,  so  data  on  the  timing  of 
submissions  passing  through  the  [)hases  can  he  collect(‘d 
hy  automatic  means.  Phases  0.5  through  07  are  manual 
processes,  so  manual  means  must  be  employed  to  <()llect 
data  on  the  titne  of  their  initiation  and  termination. 

For  request  out|)ut.  none  of  these  [)hases  is  executed, 
hut  for  other  cases  at  least  0,3  and  04  are  performed. 

Queue  for  Iransmission — Outjuit  remains  in  this  (|ueue 
until  it  can  be  transmitted.  If  the  output  is  to  he 
printed  or  otherwise  [)rocessed  at  the  central  site,  it 
does  not  enter  this  ijueue. 

transmission — Transmission  time  is  determined  mainly 
by  the  volume  of  outjmt  and  the  capacity  of  the 
communication  line.  In  some  cases  transmission  and 
printing  are  performed  simultaneously;  in  this  case 
phase  04  (Output  Process)  is  identical  with  the 
transmission  phase,  and  the  duration  is  at  least  the 
longer  of  the  two. 

Queue  for  Output  Process — Output  [)rocessed  at  the 
central  site  nearly  always  enters  this  (jueue  in 
modern  systems.  In  addition,  output  transmitted  to 
the  remote  site  may  enter  this  phase  if  transmission 
and  printing  are  se[)arate  phases. 

Output  Process — All  output  goes  through  this  phase  in  its 
production.  In  some  cases  this  output  process  may 
be  (juite  complex  as  forms  are  changed,  outjmt  is 
written  to  floppy  disk,  and  a  printer  operates. 

Queue  for  Courier — If  output  is  printed  at  the  central  site, 
it  must  be  transported  to  the  remote  site  (even  if 


Figure  .3.  Output  [ihases 
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the  central  site  and  the  remote  site  are  |)hy&ically 
(juite  close).  This  queue  occurs  between  the  time 
the  output  is  produced  ami  the  time  the  courier 
removes  it  for  transfer  to  the  remote  site. 

Courier  rransfer — The  actual  transfer  of  the  ontjiut  from 
the  central  site  to  the  remote  site  may  he  hy  special 
courier,  a  routine  courier,  or  the  m.ail.  The  titne  for 
this  phase  is  sometimes  quite  variable,  and 
sometimes  is  several  times  longer  than  all  the  rest 
of  the  [irocessing. 

Output  Return — When  the  RBS  provider  supplies 
personnel  to  operate  the  remote  batch  device,  the 
consumer  does  not  obtain  the  output  until  these 
personnel  return  the  output,  dhe  duration  of  this 
phase  should  be  small. 


Consumers  need  not  be  concerned  with  the  vast 
majority  f)f  the  16  phases  listed  for  Input,  Processing,  and 
()ut|iut.  They  are  usually  oidy  interested  in  two  or  three  of 
them — the  phase  in  which  they  input  their  submissions, 
ami  the  last  one  or  two  [>hases  in  which  the  work  is 
completed.  Given  the  various  input,  processing, 
and  output  alternatives,  the  submission  and 
completion  phases  must  be  carefully  defined  for 
metrics  to  be  clear.  This  definition  is  necessary  to 
eliminate  ambiguity  in  employing  the  metrics  for 
evaluating  the  remote  batch  service.  Figure  4  gives 
an  overview  to  aid  in  this  definition. 
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3.  Consumer-Oriented  Remote  Batch 
Service  Metrics 

Consumer-oriented  metrics  measure  the  KBS  service 
characteristics  that  are  of  concern  to  the  consumer.  The 
average  utilization  of  computer  system  resources  is  of  little 
interest  to  the  consumers  of  a  remote  hatch  service,  except 
to  the  extent  that  it  influences  the  level  of  service  they 
receive.  Consumers  are  interested  in  whether  the  remote 
batch  service  is  available  when  they  need  it,  whether  it 
operates  without  interruption,  whether  it  provides  their 
results  when  they  need  them,  and  whether  the  outputs  are 
correct.  These  attributes  are  termed  availability,  reliability, 
timeliness,  and  correctness.  They  are  derived  from  the 
functional  phases  of  the  KBS  process  that  are  described  in 
section  2.  The  consumer  is,  of  course,  concerned  with  the 
overall  KBS  process  and  how  it  affects  delivered  service. 
I’he  provider  must  consider  the  detailed  components  of  the 
process  in  efforts  to  improve  services  or  diagnose  service 
problems. 

3.1  Availability 

The  availabilitv  of  a  remote  hatch  service  must  be 
specified  in  terms  meaningful  to  the  consumer.  If  the 
consumer  walks  up  to  a  remote  batch  device  that  is  not 
operating,  then  the  service  is  unavailable.  How  well  the 
central  site  or  other  remote  batch  devices  may  be 
processing  other  jobs  is  unimportant.  Therefore, 
availability  must  be  specified  for  each  remote  site 
individually. 

Availability  can  be  specified  in  terms  of  the  portion  of  a 
selected  time  interval  (e.g.,  8:00  a.m.  to  5:30  p.m.)  during 
which  the  remote  site  is  capable  of  performing  its 
functions  such  as  accepting  submissions  and  producing 
output.  If  a  metric  is  required  for  the  RBS  as  a 
whole,  the  appropriate  one  is  the  portion  of  all 
specified  remote  sites  that  meets  the 

requirements  for  availability. 

The  interpretation  of  the  term  “availability”  varies 
across  different  types  of  remote  sites.  For  example,  a 
remote  site  that  is  operated  by  provider  personnel  is 
available  when  the  operator  is  there  to  receive  submissions 
although  another  consumer  may  also  be  there  at  the  same 
time.  If  the  remote  site  is  consumer  operated,  then 
availability  implies  that  the  remote  batch  device  is  able  to 
communicate  with  the  central  site  computer.  In  this  case, 
availability  includes  consideration  of  the  central  site 
computer  and  the  telecommunications  facilities.  If  any 
component  is  unavailable,  the  RBS  is  unavailable. 

Measurement  of  availability  must  be  performed  with 
recognition  of  the  tendency  of  consumers  to  emphasize 
periods  of  unavailability.  If  consumers  find  a  remote  batch 
device  inoperable  (perhaps  due  to  a  failed  communication 
line),  they  will  likely  make  repeated  attempts  to  use  it  in 


the  following  few  minutes.  Iherefore,  the  |)ortion  of 
attempts  that  fail  will  he  far  higher  than  the  portion  of 
time  that  the  device  is  actually  inoperable. 

In  some  remote  sites  the  input  and  the  output 
eijuipment  can  fail  independently  of  each  other.  In  these 
cases  the  provider  and  consumer  must  decide 
whether  they  tvish  to  specify  different  availability 
criteria  for  input  and  output,  or  to  specify  the 
required  availability  of  BOTH  capabilities. 

If  a  remote  hatch  device  is  able  to  begin  an  input,  but 
then  fails  during  processing  of  the  submission,  its 
computed  availability  will  be  affected  by  the  failure. 
However,  the  effect  on  the  consumers  will  be  larger  than  if 
the  failure  condition  occurred  during  a  period  when  the 
device  was  not  active.  In  this  case  the  consumer  must 
determine  the  status  of  the  submission,  output  and  the 
system  before  reprocessing  the  submission.  This  [irohlem 
involves  reliability  as  well  as  availability. 

3.2  Reliability 

The  reliability  of  a  remote  batch  service  includes  the 
reliability  of  the  central  site  to  the  extent  that  a  failure 
there  interrupts  operations  at  the  remote  batch  device.  The 
reliability  of  an  RBS  involves  the  ability  of  the  remote 
batch  device  to  operate  without  the  consumer’s  corrective 
intervention  through  a  complete  input  or  output  operation. 
For  the  consumer  operated  remote  batch  device,  corrective 
actions  might  include  restarting  a  transmission,  reloading  a 
card  deck,  rekeying  a  command,  reinitiating  a 
communication  line,  or  other  actions  outside  normal  ones 
such  as  loading  a  new  box  of  paper.  With  the  increase  in 
sophistication  of  remote  batch  devices,  many  of  these 
actions  can  be  undertaken  automatically  when  necessary. 
Automatic  correction  results  in  increased  reliability.  When 
the  remote  batch  device  is  provider  operated,  consumer 
intervention  would  be  required  and  possible  in  much  more 
limited  circumstances. 

A  variety  of  different  capabilities  may  be  associated 
with  remote  batch  devices  connected  to  a  remote  batch 
service.  For  example,  a  device  might  accept  keyed  input 
for  storage  on  a  floppy  disk,  transmit  the  -information 
automatically  at  a  specified  time,  receive  output  back  onto 
the  floppy  disk,  and  subsequently  print  it.  When  each  of 
the  operations  performed  by  a  remote  terminal 
device  has  a  different  set  of  hardware  that  can 
impact  reliability,  the  required  level  of  reliability 
can  be  specified  separately  for  each.  Some  remote 
devices,  however,  fail  as  one  unit.  When  this 
happens,  a  single  metric  for  reliability  should  be 
employed  for  the  device. 

Computation  of  reliability  of  the  different  operational 
modes  of  a  remote  batch  device  involves  determining  the 
total  time  that  the  device  successfully  operates  for  each 
mode,  and  the  number  of  times  that  corrective  consumer 
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action  is  required  during  that  operation.  A  ratio  of  average 
successful  operation  time  to  number  of  required  corrective 
actions  can  be  calculated.  If  the  remote  batch  device  halts 
operation  after  a  failure  and  does  not  restart  until 
corrective  action  is  taken,  the  time  between  the  failure  and 
the  corrective  action  must  not  be  included  in  tbe  time  of 
successful  operation.  Elapsed  time  is  often  reported  as 
Mean  Time  Between  Failures  (MTBF). 


3.3  Timeliness 

Timeliness  is  usually  equated  to  “turnaround,”  i.e.  tbe 
elapsed  time  between  the  moment  of  submission  and  the 
instant  that  output  is  returned.  Timeliness  can  also  be 
specified  in  terms  of  a  deadline,  when  completion  of  the 
processing  of  submissions  is  required.  However,  if  only  the 
deadline  for  completion  is  specified,  the  providers  may 
have  an  impossible  job.  The  consumers  must  input  their 
submissions  early  enough  to  allow  processing  to  occur 
before  the  deadline,  so  agreements  regarding  service 
level  in  terms  of  deadline  must  include 
specification  of  submission  time. 

A  computer  installation  that  can  always  fulfill 
all  its  timeliness  objectives  probably  has  far  too 
much  hardware.  At  some  time  in  its  life,  the  load 
is  certain  to  peak  to  an  unusual  level  and  result 
in  underfulfillment  of  some  consumers’ 
objectives  unless  a  luxurious  surplus  of  capacity 
exists.  The  normal  case  is  for  a  remote  batch 
service  to  fulfill  reasonable  goals  most  of  the 
time,  hut  to  fail  some  portion  of  the  time. 
Therefore,  timeliness  objectives  should  be  stated 
as  the  portion  of  instances  in  which  specified 
timing  objectives  are  met. 

Specification  or  measurement  of  a  remote  batch 
system’s  timeliness  requires  the  identification  of  events 
signifying  the  submission  time  and  the  completion  time. 
Special  hardware  or  software  may  be  needed  for  this 
identification. 

3.3.1  Submission  Time 

The  time  of  submission  is  the  start  of  any  turnaround 
measure.  In  general,  the  submission  time  is  tbe  instant 
that  the  consumer  transfers  control  of  the  submission  over 
to  the  BBS  provider.  The  three  input  environments  have 
different  events  signifying  submission  times: 

Provider  Operated.  When  the  provider  supplies  personnel 
to  operate  the  remote  batch  device,  the  submission 
time  is  the  instant  when  the  consumer  turns  over 
the  media  with  the  submission.  Because  it  is  usually 
small,  tbe  delay  when  the  operator  is  busy  with 
another  consumer  is  ignored. 


Consumer  Operated.  When  the  remote  batch  device  is  a 
conventional  one  that  accepts  card  or  other  physical 
media  input,  the  submission  time  is  the  instant  that 
the  first  record  is  read. 

Interactive  Terminal  Originated.  When  the  remote  batch 
device  is  a  terminal  also  employed  for  interactive 
services,  the  submission  time  is  the  instant  that  the 
RETURN  key  or  the  ENTER  key  is  depressed  on 
the  terminal. 

In  some  remote  batch  systems  more  than  one  of  these 
types  of  input  may  be  employed.  When  more  than  one 
type  of  input  is  employed,  the  applicability  of 
criteria  must  be  specified  by  type. 

3.3.2  Completion  Time 

Several  types  of  output  may  be  employed  for  one 
consumer.  In  such  cases  the  choice  of  completion  time  may 
make  a  dramatic  difference  in  the  value  of  the  metric.  For 
example,  a  common  approach  is  to  provide  summary 
output  to  a  terminal  used  for  interactive  output  on  an 
immediate  basis  with  more  extensive  output  printed 
subsequently.  The  delay  in  printing  the  extensive  output 
may  exceed  the  time  between  submission  and  initial 
output,  so  choosing  one  completion  time  rather  than  the 
other  could  change  the  turnaround  time  by  a  large  factor. 
When  two  or  more  output  devices  are  employed, 
timeliness  metrics  must  identify  which  devices 
are  applicable  to  each. 

Output  from  the  remote  batch  service  may  be 
immediately  transmitted  to  the  remote  batch  device,  or  it 
may  be  held  until  requested  by  the  consumer.  The 
completion  time  is  associated  with  the  final  output 
operation.  For  immediate  output  the  completion 
time  is  the  instant  when  the  last  record  of  output 
is  printed.  For  request  output  the  completion 
time  is  the  instant  when  the  user  is  notified  of 
output  availability,  or  when  the  last  record  of 
output  is  available  for  transmission  upon 
request. 

In  some  cases  the  objective  of  a  submission  is  to  update 
a  database  rather  than  to  produce  output.  For  database 
updates  the  completion  time  is  the  instant,  upon 
completion  of  the  database  update,  when  the 
revised  data  can  be  accessed.  When  both  a  database 
update  and  a  set  of  outputs  have  separate  timeliness 
objectives  for  a  single  submission,  the  two  criteria  should 
be  specified  separately. 

3.3.3  Intermediate  Times 

Consumers  may  only  be  interested  in  the  level  of 
service  received,  but  their  actions  can  significantly 
influence  the  level  received.  If  the  consumers  served  by  a 
single  remote  batch  device  increase  their  volume  of  output. 
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for  instance,  they  will  likely  experience  degraded 
timeliness.  The  providers  will  be  interested  in  the  cause  of 
this  degradation  because  they  may  need  to  take  corrective 
action.  The  consumers  may  also  be  interested,  however, 
because  simple  actions  on  their  part  (e.g.,  reducing  output 
volume)  may  result  in  improved  timeliness  without 
increased  cost.  Each  of  the  stages  of  flow  through  the  RBS 
scenarios  in  the  previous  section  can  be  measured  and 
reported  to  aid  in  determining  whether  changes  are 
appropriate. 

The  functional  definitions  of  submission  and  completion 
times  provided  above  have  general  applicability  because 
they  describe  the  effects  that  the  consumer  observes  about 
the  RBS  rather  than  the  techniques  employed  in  its 
implementation.  The  definition  of  intermediate  times, 
however,  is  dependent  on  the  particular  implementation 
characteristics  of  the  remote  batch  system.  For 

maximum  utility,  intermediate  times  in  the  flow 
of  submissions  must  be  defined  for  each  remote 
batch  system  as  a  special  case.  These 
intermediate  times  should  be  employed  in 
performing  studies  on  timeliness. 

3.4  Correctness 

The  remote  batch  service  consumer  has  the  same 
problems  related  to  correctness  as  the  on-site  batch 
consumer.  In  addition,  there  are  errors  which  could  be 
introduced  by  the  telecommunications  facility  and  the 
remote  batch  device.  Undetected  transmission  errors  may 
cause  software  errors  and  compromise  the  correctness  of 
the  data  processing.  Identifying  the  source  of  such  errors 
is  practically  impossible. 

The  current  state-of-the-art  for  measuring  correctness 
largely  depends  on  rerun  rates  for  determining  correctness 
levels.  Although  internal  balancing  controls  can  ease  this 
job  considerably,  identifying  a  submission  as  a  rerun  due 
to  provider  actions  usually  requires  human  review  of 
output.  This  identification  is  frequently  subject  to  dispute 
between  the  providers  and  consumers  because  of  differing 
interpretations  of  events  and  the  potential  impact  on 
payments  from  the  consumers  to  the  providers.  Therefore, 
agreement  is  needed  on  the  mechanism  for 
determining  that  a  job  must  be  rerun  because  of 
the  provider's  actions. 

Specifications  for  correctness  have  to  take  into  account 
problems  that  might  arise  as  a  result  of  the  kind  of  service 
provided.  For  example,  the  operational  problems  and  the 
difficulties  with  obtaining  the  correct  data  are  sometimes 
less  important  than  inadequate  disk  space  (or  other 
resource)  for  the  execution  of  a  particular  submission. 
Some  types  of  problems  (e.g.,  support  software  such  as 
compilers)  may  be  far  more  difficult  to  detect  or  correct, 
and  may  have  more  important  effects  (e.g.,  a  late  payroll), 
than  others  (e.g.,  one  more  test  run).  Therefore,  required 


levels  of  service  should  be  specified  in  terms  of 
the  portion  of  all  submissions  that  may  be 
affected  by  these  conditions. 

I’o  the  consumer,  detected  failures  in  correctness  are 
costly  to  the  degree  to  which  they  affect  timeliness. 

4.  RBS  Measurement  Methodology 

The  four  attributes  applicable  to  RBS — availability, 
reliability,  correctness  and  timeliness — cannot  be 
measured  equally  well.  Techniijues  for  measuring 
timeliness  are  the  best  developed. 

4.1  Timeliness  Measurements 

The  measurement  methodology  which  can  he  afiplied  to 
remote  batch  service  depends  on  the  scenario  which  best 
describes  the  way  in  which  the  RBS  is  provided  to  the 
consumer.  As  pointed  out  in  section  3,  different  activities 
and  queues  exist  in  different  scenarios.  Consequently,  the 
consumer’s  perception  of  the  quality  of  service  differs. 
This  differentiation,  in  turn,  requires  that  different 
measurement  methodologies  be  employed  according  to  the 
way  in  which  the  RBS  and  the  consumer  interact. 

4.1.1  State-of-the-Art 

The  most  common  way  of  measuring  remote  batch 
service  is  by  the  central-site  computer  which  is  providing 
the  service.  Times  are  associated  with  a  joh  when  it  enters 
the  central-site  computer  and  when  it  leaves.  There  is  great 
variability  among  operating  systems  as  to  the  exact  step  in 
the  queuing  of  input  and  output  associated  with  these 
times.  In  many  cases,  this  information  is  not  available  and 
can  only  be  determined  by  a  systems  programmer  reading 
the  instructions  in  the  front-end  and/or  central  processor 
concerned  with  communications. 

For  consumer  operated  and  interactive  terminal  input, 
measurement  of  input  times,  and  of  request  and  immediate 
output  times,  can  be  done  by  the  central  site  computer. 
Measurement  within  the  central-site  computer  is 
conditional  on  ignoring  delays  within  the  data 
communications  system.  The  validity  of  this  assumption 
appears  to  be  based  on  the  relative  brevity  of  data 
transmission  as  compared  to  processing.  When  there  is  an 
interruption  to  data  transmission  due  to  failure  of  some 
component  in  the  data  communications  system,  this 
assumption  is  violated,  but  the  central-site  computer  may 
not  be  programmed  to  modify  its  timing  accordingly. 

Another  way  to  measure  RBS  is  to  use  a  “job  submittal 
card”  with  each  job  submitted  for  remote  batch  service. 
Along  with  other  functions,  the  job  submittal  card  provides 
a  place  for  a  time  stamp  when  the  job  enters  and  leaves 
the  system. 

In  a  provider  operated  remote  batch  service,  the 
consumer  could  time  stamp  the  job  submittal  card  when 
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passing  the  job  to  the  attendant.  The  attendant  could  time 
stamp  the  card  when  output  was  ready  to  be  claimed  by 
the  consumer.  In  the  case  of  a  consumer  operated  remote 
batch  service,  the  time  stamp  could  be  em[)loyed  for  job 
submission  only. 

4.1.2  Extensions  to  the  State-of-the-Art 

A  printing  clock  could  be  added  to  the  printer  at  a 
remote  batch  site  which  would  automaticallv  print  the  time 
on  the  output.  This  timer  could  be  set  to  print  at  a  fixed 
interval  which  would  provide  satisfactory  granularity.  This 
timer  would  be  useful  in  the  case  of  a  consumer  operated 
remote  batch  service;  the  time  stamp  closest  to  the  end  of 
the  printout  would  identify  when  the  output  was  ready  for 
the  consumer. 

If  the  remote  batch  device  could  associate  a  unique 
identifier  with  each  job,  it  could  participate  in  the  timing. 
The  presence  of  such  a  unique  identifier  is  dependent  on 
the  operating  system;  it  is  not  universally  available.  When 
a  unique  job  identifier  was  available,  the  remote  batch 
device  could  associate  a  time  with  the  beginning  of  job 


input  and  the  end  of  job  output.  This  timing  would  be 
useful  in  the  case  of  consumer  operated  remote  batch 
service. 


4.1.3  Data  Acquisition  and  Analysis 

The  coverage  of  the  methodology  described  above  to 
the  scenario  descriptive  of  the  remote  batch  service  is 
shown  in  Tables  1  and  2.  Any  acceptable  pair  of 
methodologies  may  be  selected  as  the  basis  for 
analysis,  one  for  input  and  another  for  output. 
The  need  to  perform  statistical  analysis  on  the 
measured  data  strongly  favors  a  measurement 
methodology  which  provides  its  data  in  machine 
processible  form.  Thus  data  acquired  in  the  host 
computer  and  in  the  remote  batch  equipment  is  well  suited 
for  analysis. 

If  the  manual  time  stamp  could  also  provide 
information  in  machine  readable  form,  its  potential 
usefulness  would  be  increased.  If  the  stamp  characters 


Table  1.  Inp  ut  Measurement  Methodnlogy  Related  to  Service  Scenario 


Service  Scenario 

By  Host 
Computer 

Manual  Time 

Stamp 

.lob  Timing  In 

Remote  Batch  Device 

Provider  Operated 

N:5 

C:2,8 

N:5 

Consumer  Operated 

C:6 

C:2,6,7,8 

A:8 

Interactive  Originated 

A 

C:2.8 

X:4 

Table  2.  Output  Measurement 

Methodology  Related  to  Service  Scenario 

Service  Scenario 

By  Host 
Computer 

Manual  Time 

Stamp 

Printout 

Time  Stamp 

Job  Timing  In 

Remote  Batch  Device 

Delivered 

N;l..3 

A;2.8.9 

N:1.3 

X:4 

lnime<liate  (provider  operated) 

N:l 

A:2,8.9 

N:,5 

N:5 

Immediate  (consumer  operated) 

C:2,6,7 

.A:2,8,9 

.A:2.8 

A:8 

Request 

C:6 

A:2,8,9 

A:2.8 

A:8 

Key 

A  -  Acce[ital)le. 

C  -  Conditionally  acceptable. 

N  •  Not  acceptable. 

X  •  Not  applicable. 

1  ■  Excludes  time  outside  host  (central  site)  computer. 

2  -  If  a  means  exists  for  automating  data  analysis. 

'i  Excludes  [leriod  when  output  is  not  available  to  consumer. 

4  -  Remote  batch  device  not  involved. 

5  •  Excludes  time  outside  computer/communications  system. 

6  ■  If  there  is  no  significant  delay  in  transmission. 

7  -  It  there  is  no  signilicant  delay  between  time  stamping  and  transmission. 

8  -  If  a  means  exists  for  asstx  iating  input  and  output  of  each  jid). 

9  •  Time  stam[>  applied  when  output  becomes  available  to  consumer. 
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were  optical  character  recopjnition  (OCR)  or  magnetic 
character  recognition  (MICK),  then  machine  input  would 
be  possible.  Another  possibilitv  is  for  the  time  clock  to 
punch  the  job  submittal  card  so  that  it  could  be  read  into  a 
computer. 

Data  acquisition  must  be  planned.  In  fact,  data 
acquisition  and  analysis  must  be  specified  in 
advance.  If  contractual  guarantees  of  service  are 
employed,  these  specifications  must  be  part  of 
the  contract.  Data  accjuisition  could  be  performed  by 
sampling  actual  KBS  utilization.  The  sanq)ling  interval  and 
rate  would  have  to  be  selected  so  as  to  be  statisticallv 
valid.  Benchmarking,  [5|  especially  the  running  of  a 
specified  benchmark  under  normal  operating  conditions,  is 
another  data  actjuisition  technique  [6|. 

Meaningful  reports  describing  remote  batch  service  will 
require  statistical  analysis.  The  number  of  observations  to 
be  taken  must  be  sufficiently  large  to  be  statistically 
significant.  Based  on  the  distribution  of  this  data, 
statistical  descriptors  such  as  mean  and  standard  deviation, 
or  percentile  statistics  such  as  median,  90  percentile,  etc., 
must  be  selected.  Then  the  data  must  be  analyzed.  A 
discussion  of  statistical  methods  is  beyond  the  scope  of 
this  document.  The  statistical  criteria  and  sample 
size  should  be  chosen  in  advance  of  any 
measurement  of  service  quality. 

4.2  Other  Measurements 

The  remaining  attributes  (correctness,  availability,  and 
reliability)  are  more  difficult  to  quantify  automatically  than 
timeliness.  The  discussions  accompanying  the  definition  of 
these  metrics  included  means  of  measurement,  which  are 
summarized  here. 

Correctness  determination  requires  human  intervention. 
There  is  no  algorithm  which  can  differentiate  between 
correct  and  incorrect  results  as  discussed  herein.  Once 
manually  identified,  incorrect  runs  may  be  repeated  to 
obtain  correct  results.  The  time  or  cost  or  delay  of  such 
reruns  may  be  tabulated  as  a  measure  of  correctness. 

Availability  and  reliability  of  KBS  can  involve  failure  of 
the  remote  batch  device.  It  is,  therefore,  impractical  to  rely 
on  that  device  to  collect  availability  statistics.  In  some 
scenarios  the  KBS  extends  beyond  the  remote  batch 
device;  this  eliminates  the  possibility  of  using  the  KBS  for 
data  collection.  Manual  means,  such  as  logs,  remain  as 
data  sources  for  measures  of  availability  and  reliability. 
The  process  can  be  partially  automated  by  use  of  machine 
sensible  forms  to  record  failure,  unavailability,  and 
completion  of  repair. 

5.  RBS  Measurement  Environment 

Choose  a  monitor;  collect  some  data;  wonder  what  to  do 
with  the  data.  This  popular  approach  has  the  advantage  of 


being  obvious,  and  the  disadvantage  ol  being  worthless. 
One  of  its  fatal  Haws  is  that  it  collects  data  on  an 
undefined  environment.  The  situation  it  examines  is  of 
fleeting  duration,  and  will  likely  never  occur  again  in 
[)recisely  the  same  way.  The  differences  may  not  be 
significant — or  they  may  be  significant — but  (piite 
probably  no  one  will  be  able  to  determine  whether  the 
changes  are  significant  or  irrelevant,  d'he  alternatives 
available  to  the  analyst  are  many,  the  successful  ones 
include  choosing  objectives  prior  to  picking  a  measurement 
ap[)roach,  and  defining  the  environment  prior  to  taking 
measurements. 

After  choosing  objectives  carefully,  the  analyst  can 
decide  whether  to  control  the  loading  and,  if  so,  whether 
these  controls  should  be  total  or  partial.  In  jiartial  control 
the  normal  situation  is  usually  allowed  t(j  exist,  but  some 
additional  load  is  added  with  known  characteristics.  In 
total  control  the  remote  batch  system  is  idle  except  for  a 
totally  artificial  load. 

As  more  control  is  exercised  over  the  load,  a  decline 
may  occur  in  the  degree  to  which  the  resulting  situation 
represents  the  normal  situation.  However,  increased 
control  improves  the  ability  of  the  analyst  to  interpret  the 
data;  in  many  cases  it  “improves”  ability  from  an 
impossible  challenge  to  a  possible  one  due  to  decreased 
random  variability.  The  analyst,  therefore,  must  choose 
how  much  control  should  be  exercised  so  that  variability 
reduction  can  be  traded  off  against  representativeness.  The 
trade-off  decision  is  dependent,  of  course,  on  the  objectives 
of  the  analysis. 

5.1  Measurement  Under  Uncontrolled  Conditions 

The  most  obvious  approach,  the  one  that  can  usually  be 
implemented  most  readily,  and  the  one  that  is  usually  the 
logical  first  step  in  an  analysis  effort,  is  measurement  of  an 
uncontrolled  loading  situation.  Most  installations  keep 
some  historical  data  on  their  performance,  so  measurement 
of  the  current,  uncontrolled  situation  can  be  compared 
with  the  past  performance.  Some  of  the  potentially 
valuable  insights  that  can  come  from  such  an  analysis 
involve: 

The  degree  to  which  performance  varies: 

The  types  of  remote  submissions  commonly  input  to  the 
central  site; 

Generally  achievable  levels  of  reliability,  availability, 
timeliness,  and  correctness; 

The  dependability  of  the  measurement  devices. 

Measurement  under  uncontrolled  conditions  is,  of 
course,  the  most  widely  employed  technique  for  obtaining 
information  in  support  of  [)erformance  management.  A 
performance  management  system  usually  attempts  to 
control  both  the  load  and  the  performance  in  response  to 
that  load.  Both  of  these  reijuire  measurement  of  the 
natural  environment.  However,  the  results  ol  such 
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measurements  must  be  interpreted  carefully  because  of  the 
confusing  variety  of  effects  that  can  occur  and  the 
limitations  of  the  measurement  tools  commonly  employed. 

The  standard  job  accounting  system  from  computer 
vendors  sometimes  fails  to  report  the  presence  of  certain 
submissions  due  to  the  operating  system  failing  prior  to 
the  completion  of  the  submission’s  processing.  Since  the 
job’s  resource  demands  may  only  be  reported  at  job 
completion,  the  analyst  may  know  that  a  submission  was 
not  processed,  but  not  know  that  it  consumed  a  major  part 
of  the  computer  for  several  hours.  Therefore,  the  analyst 
concludes  that  the  service  has  become  unaccountably  sick 
and  fails  in  the  analysis.  Special  care  should  be 
exercised  to  detect  the  presence  of  submissions 
that  consume  unusually  large  quantities  of 
resources  in  an  uncontrolled  loading  en¬ 
vironment. 

In  addition  to  anomalous  conditions,  most 
remote  batch  services  experience  peaks  in 
demand  on  a  periodic  basis.  Projections  from 
periods  of  low  demand  to  periods  of  high  demand 
are  typically  unreliable.  Characterization  of 
remote  batch  service  performance  often  requires 
examination  at  both  peaks  and  valleys  of 
demand.  Since  many  RBSs  also  have  an  annual  peak, 
the  analyst  may  conclude  that  measurement  of  the 
uncontrolled  environment  requires  examination  of 
operations  over  a  full  year.  This  excessively  expensive 
approach  is  seldom  needed.  Instead,  the  analyst  may  need 
to  refine  the  objectives  or  to  employ  some  degree  of 
control  over  the  load. 

5.2  Measurement  Under  Partial  Control 

Partial  control  of  a  remote  batch  service  may  be 
employed  to  ensure  that  a  few  submissions  with  unusually 
large  resource  demands  do  not  dominate  measurement 
results,  to  determine  the  effects  of  heavier  loading  such  as 
year  end  processing,  or  to  investigate  specific  effects  that 
might  decrease  availability  such  as  a  new  software 
package.  These  objectives  can  usually  be  achieved  with 
only  a  minimal  impact  on  the  services  to  normal 
consumers,  so  the  cost  of  the  investigation  is  far  less  than 
in  the  case  of  total  control.  Because  some  random 
variability  can  be  controlled,  more  confidence  can  usually 
be  placed  in  conclusions  about  the  effects  of  a  change. 

Implementation  of  partial  control  may  make  obvious  to 
consumers  that  the  service  is  not  in  a  normal  state.  This  is 
particularly  true  when  the  control  is  exercised  at  the 
remote  site  (by  limiting  submissions  to  those  that  do  not 
have  extensive  input,  for  example).  Consumers  may  alter 
their  normal  activities  in  response  to  their  recognition  that 
different  service  is  being  provided.  Elapsed  times  through 
intermediate  measurement  points  may  become  very 
different  when  this  occurs.  Measurement  under 


partially  controlled  conditions  may  require 
measurement  under  uncontrolled  conditions  in 
order  to  provide  comparison  values  for  detection 
of  uncontrolled  changes. 

Artificially  loading  a  remote  batch  service  is  usually  an 
easy  task.  The  analyst  need  only  input  from  one  or  more 
remote  batch  devices  a  few  special  submissions.  The 
submissions  must  be  carefully  chosen  or  constructed,  but 
after  that  the  mechanics  for  causing  the  loading  are 
simple.  Since  submission  is  so  easy,  the  analyst  may  find 
that  submitting  a  small  test  job  and  measuring  its 
turnaround  time  provides  an  easy  way  to  determine  the 
timeliness  of  the  BBS. 

5.3  Measurement  Under  Total  Control 

The  effects  of  any  uncontrolled  loading  in  a  remote 
batch  system  can  confuse  the  analysis.  The  magnitude  of 
operating  system  overhead,  for  example,  can  be  blamed  for 
delays  in  processing  by  one  consumer  while  another 
blames  loading  on  a  shared  communication  line.  Both — or 
neither — could  be  correct,  but  determination  of  the  correct 
answer  is  difficult  unless  all  perturbing  effects  are 
eliminated.  This  elimination  can  be  achieved  only  by 
gaining  total  control  over  the  load. 

Measurement  under  total  control  causes  the  remote 
batch  service  to  be  unavailable  to  consumers  during  the 
period  of  the  experiment.  Therefore,  the  experiment  will 
be  expensive  in  execution  and  require  considerable  effort 
to  define  objectives,  procedures,  and  analysis  approach 
before  its  initiation.  Measurement  under  total 
control  is  available  to  the  provider  of  the  RBS.  It 
should  be  pursued  only  if  partial  control  is 
inadequate  and  only  after  thorough  design  of  the 
experiment  [1,7]. 

The  objective  of  the  experiment  has  considerable 
bearing  on  its  design.  A  test  to  determine  the  effect  of 
some  hardware  or  software  change  at  the  central  site  would 
be  quite  different  than  a  test  to  determine  why  one  remote 
batch  site  was  receiving  very  slow  turnaround  compared  to 
other  remote  batch  sites.  In  the  design  of  the  experiment, 
the  analyst  must  decide  on  the  technique  for  loading  the 
remote  batch  service.  If  the  experiment  is  to  investigate 
effects  localized  in  the  central  site,  the  analyst  may  find 
that  the  situation  can  be  treated  like  a  conventional  batch 
system  and  load  a  series  of  jobs  through  any  convenient 
input  device  (e.g.,  tape  or  punched  cards).  This  approach 
may  ease  control  by  putting  the  analyst  in  the  computer 
room  where  he  or  she  can  input  all  artificial  submissions 
and  also  ensure  that  any  special  operational  procedures  are 
followed. 

Alternatively,  the  analysts  may  find  their  jobs  easier  if 
they  load  submissions  from  a  remote  batch  device.  This 
may  actually  be  tbe  only  mechanism  that  will  operate 
successfully  in  some  systems  where  recognition  of  the 
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input  device  is  important  in  determining  the  way  the 
system  performs.  In  addition,  the  objectives  of  the 
experiment  may  require  that  the  effects  of  loading  on  the 
remote  batch  devices  need  to  be  determined. 

6.  Conclusions 

6.1  Required  Technologieal  Advances 

The  technology  for  measuring  remote  batch  services  is 
deficient  in  several  respects.  In  some  cases  the  needed 
advances  represent  rather  minor  extensions  of  existing 
technology  and  can  be  accomplished  rapidly.  In  other 
cases  extensive  research  is  required  to  correct  the 
deficiency.  This  section  describes  needed  improvements  in 
measurement  technology  for  the  four  metrics  applicable  to 
RBS. 

6.1.1  Availability 

When  a  communication  line  is  continuously  active 
between  the  remote  site  and  the  central  site,  availability 
can  be  determined  by  having  the  remote  batch  device 
automatically  attempt  to  send  and  receive  trivial  messages. 
Although  this  capability  is  not  currently  a  standard  part  of 
remote  batch  device  software,  its  addition  should  not 
present  any  overwhelming  problems. 

However,  when  dial-up  lines  are  used  for  transmission, 
measurement  of  availability  is  more  difficult.  The 
automatic  dial-up  capability  of  some  terminals  could  be 
invoked  repeatedly  resulting  in  extra  communication 
charges  to  compute  a  number  for  availability.  When  the 
communication  line  must  be  manually  dialed,  determining 
availability  through  repeated  pseudo-use  would  simply  be 
unrealistic.  Instead,  research  is  needed  to  determine 
whether  the  individual  failure  rates  of  remote  batch 
devices  and  central  sites  can  be  employed  to  estimate  total 
remote  batch  service  availability  for  individual  remote 
sites.  Some  of  the  issues  that  require  resolution  are: 

Availability  of  communication  ports. 

Failure  of  some  shared  ports  at  a  central  site. 

Failure  of  the  communication  capability  of  the  remote 
batch  device. 

Automatic  transmission  of  failure  information  to  the 
central  site. 

Simultaneous  failure  of  remote  and  central  equipment. 

One  potential  approach  to  determining  availability  is  to 
provide  hardware  devices  that  simulate  a  central  site  when 
connected  to  a  remote  batch  device.  Although  mechanical 
devices  probably  could  not  be  checked  with  such  a  device, 
the  bulk  of  electronic  equipment  could  be  examined  every 
few  seconds. 


6.1.2  Reliability 

Inclusion  of  minimal  data  collection  ca[)abilities  in  the 
central  computer  and  the  remote  batch  device  should  be 
adequate  for  determining  reliability.  Although  these 
capabilities  are  not  now  included  in  most  systems,  their 
development  should  not  be  difficult. 

6.1.3  Timeliness 

Three  primary  technological  inadequacies  limit  the 
utility  of  timeliness  determinations  in  remote  batch 
services.  They  are: 

Sophisticated  remote  batch  devices  may  create  output  at 
a  different  time  than  the  central  site’s  computer  has 
recorded.  Collection  of  data  about  submission  and 
completion  times  should  be  automatic  at  the  remote 
batch  device. 

The  beginning  and  ending  of  data  base  updates  is 
frequently  difficult  to  determine.  These  times  should  be 
recorded  automatically  and  identified  in  a  manner  so 
that  turnaround  and  appropriate  metrics  can  be 
employed. 

Data  about  submission  and  completion  times  that  are 
collected  at  the  remote  sites  do  not  automatically  find 
their  way  into  the  central  site!  Therefore,  the  important 
information  about  how  well  a  complex  automated 
system  performs  is  created  with  the  use  of  manual 
processing. 

6.1.4  Correctness 

Measurement  of  correctness  by  examination  of  rerun 
rates  is  inadequate.  In  most  cases  the  metric  is  employed 
in  a  rather  arbitrary  manner  and  cannot  be  relied  upon.  In 
some  instances  the  values  obtained  have  been  so  open  to 
question  that  installations  have  stopped  trying  to  evaluate 
correctness. 

A  significant  research  program  is  critically  needed  in 
the  area  of  operations  correctness  in  commercial  data 
processing  installations.  It  should  address  the  problems 
that  result  in  incorrectness  from  operations,  programming, 
data  input,  and  catalog  maintenance. 

6.2  What  Can  and  Should  Be  Done  Now 

Even  though  measurement  technology  could  stand  some 
improvement,  there  is  presently  a  sufficient  basis  for 
remote  batch  service  performance  management.  A  very 
modest  effort,  such  as  the  one  suggested  here,  can 
significantly  improve  relations  between  providers  and 
consumers  of  remote  batch  service.  It  can  further  provide 
each  of  them  with  metrics  by  which  first,  to  measure,  and 
second,  to  manage,  tbeir  involvement  with  RBS.  The 
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measurement  approach  presented  in  these  guidelines 
addresses  only  part  of  remote  batch  service  operation.  It 
should  be  employed  in  context  as  part  of  a  performance 
management  system. 

The  first  step  is  the  identification  of  the  provider  and 
consumer.  This  is  not  necessarily  trivial  when  dealing  with 
groups  of  consumers  who  may  belong  to  different 
organizational  units  and  who  may  deal  with  the  provider 
through  different  spokespersons. 

I'he  second  step  is  selection  of  the  scenario  descriptive 
of  how  the  consumer  obtains  RBS.  The  scenarios 
presented  in  this  report  should  serve  as  a  starting  place. 
They  may  be  sufficient,  or  may  require  development 
before  being  acceptable.  Both  provider  and  consumer 
should  agree  on  the  representativeness  of  the  scenario. 

The  third  step  is  the  selection  of  RBS  metrics.  This 
selection  must  be  done  in  concert  with  the  fourth  step, 
selection  of  measurement  methodology.  The  problem  is  to 
determine  what  should  be  measured  and  what  can  be 
measured.  This  process  can  also  identify  needed  advances 
in  measurement  methodology.  A  cost  benefit  study  can  be 
conducted  to  help  decide  whether  technology  advancement 
in  measurement  methodology  should  be  pursued.  The 
consumer  group  should  be  kept  aware  of  and  educated 
about  the  selections  of  metrics  and  methodology  and  the 
logic  which  led  to  that  selection. 


The  final  step  is  implementation.  RBS  performance 
evaluation  may  be  employed  to  settle  disputes,  to 
contractually  establish  definitions  of  satisfactory  RBS,  to 
support  negotiation  or  litigation  when  agreed-upon  service 
criteria  are  violated,  to  alert  provider  management  when 
service  levels  are  out-of-bounds,  or  to  help  consumers 
select  a  new  RBS. 
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Glossary 


BATCH  PROCESSING:  1)  T  he  processing  of  data  or  the  accomplishment  of  jobs  accumulated  in  advance  in  such  a 
manner  that  each  accumulation  thus  formed  is  processed  or  accom|)lished  in  the  same  run.  2)  The  [)rocessing  of  data 
accumulated  over  a  period  of  time.  BATCH  PROCESSING  can  be  differentiated  from  INTPHCAfiTI VE  processing. 

CENTRAL  SITE:  The  facility  maintained  by  the  PROVIDER  of  the  REMOTE  BATCH  SERVICE  to  perform  data 
processing.  This  facility  usually  includes  one  or  more  computers  with  data  storage  devices,  software,  |)ersonnel,  su[)[)ort 
services,  and  communications  facilities.  A  REMOTE  BATCH  SERVICE  may  be  supported  by  several  central  sites,  and 
these  central  sites  may  be  nodes  on  a  computer  network. 

CONSUMER:  The  CONSUMER  of  a  REMOTE  BATCH  SERVICE  employs  the  service  to  satisfy  needs  for  data 
processing.  The  CONSUMER  may  be  either  developing  programs  or  employing  data  processing  to  su[)port  other  (e.g., 
non-data  processing)  functions. 

IMMEDIATE  OUTPUT  MODE:  In  IMMEDIATE  OUTPUT  MODE  the  output  destined  for  a  particular  REMOTE  SITE 
is  transmitted  when  the  output  is  ready,  the  necessary  communications  lines  are  available,  and  the  REMOTE  BATCH 
DEVICE  is  ready  to  receive.  Output  in  IMMEDIATE  OUTPUT  MODE  may  be  ordered  by  priority,  and  it  may  be 
transmitted  to  the  REMOTE  SITE  for  queuing  in  a  REMOTE  BATCH  DEVICE  with  the  capability  to  store  the  output 
for  later  conversion  to  readable  form. 

INPUT  PROCESS:  The  process  that  consists  of  the  reception  of  data  into  the  data  processing  system,  into  a  subsystem,  or 
into  a  device.  Synonymous  with  input.  The  INPUT  PROCESS  to  a  REMOTE  BATCH  SYSTEM  includes  all  steps 
taken  to  prepare  the  SUBMISSION  for  processing. 

INPUT  TO  RBS:  The  transfer  of  control  over  a  submission  to  the  RBS  for  the  purpose  of  processing.  This  may  be  a 
manual  process  if  PROVIDER  personnel  operate  the  REMOTE  BATCH  DEVICE,  or  if  the  CONSUMER  operates  the 
REMOTE  BATCH  DEVICE.  Several  attempts  may  be  necessary  by  the  CONSUMER  to  INPUT  TO  RBS  due  to 
availability  problems  with  the  RBS. 

INTERACTIVE:  A  mode  of  operation  wherein  the  computer  and  consumer  communicate  with  each  other  through  on-line 
keyboard  terminals.  In  this  mode  of  operation,  each  unit  of  transmission  from  the  terminal  is  processed  immediately 
and  a  response  returned  to  the  consumer.  A  “dialog”  is  said  to  exist  between  the  consumer  and  computer.  Interactive 
computer  utilization  may  be  used  for  programming,  problem  solving,  inquiry,  file  update,  text  editing,  or  other  data 
processing  tasks. 

JOB:  A  group  of  tasks  prescribed  as  a  unit  of  work  for  a  computer.  A  job  begins  with  identification  of  the  consumer,  the 
source  of  authorization/funding,  and  other  information  necessary  for  operation;  and  ends  when  a  termination 
instruction,  or  equivalent,  is  executed. 

OUTPUT  PROCESS:  The  process  that  consists  of  the  delivery  of  data  from  a  data  processing  system,  from  a  subsystem, 
or  from  a  device.  In  a  REMOTE  BATCH  SERVICE  each  REMOTE  SITE  must  possess  a  capability  to  receive  output 
via  communication  lines,  but  output  may  also  undergo  media  conversion  at  the  CENTRAL  SITE  with  the  converted 
media  (usually  paper  or  microfiche)  physically  delivered  to  the  REMOTE  SITE.  Output  may,  therefore,  be  provided 
physically,  or  via  transmission  in  a  REQUEST  OUTPUT  MODE  or  in  an  IMMEDIATE  OUTPUT  MODE. 

PERFORMANCE  MANAGEMENT  SYSTEM:  A  set  of  approaches,  procedures,  and  reporting  mechanisms  designed  to 
control  a  data  processing  system  so  that  acceptable  consumer  service  can  be  reliably  provided.  Objective  data  on 
service  to  CONSUMERS,  as  well  as  usage  of  system  resources,  are  compared  with  previously  established  criteria  so 
that  actions  can  be  taken  when  performance  deviates  from  anticipated  values.  A  performance  management  system  may 
address  just  the  production  aspects  of  a  computer  service,  or  it  may  include  the  development  and  maintenance  aspects 
as  well.  For  adequacy  in  a  REMOTE  BATCH  SERVICE,  a  PERFORMANCE  MANAGEMENT  SYSTEM  must  address 
all  areas  in  which  the  PROVIDER  supplies  SERVICE. 

PROVIDER:  The  PROVIDER  of  a  REMOTE  BATCH  SERVICE  maintains  the  hardware,  software,  personnel,  support 
services,  and  communications  facilities  necessary  to  the  SERVICE.  The  PROVIDER  may  also  supply  a  communications 
network,  REMOTE  BATCH  DEVICEs,  and  personnel  at  the  REMOTE  SITEs. 

REMOTE  BATCH  DEVICE:  Equipment  with  the  capability  of  receiving  a  submission  to  RBS  in  machine-sensible  form 
for  transfer  to  a  communication  facility  and/or  the  capability  for  receiving  output  from  the  RBS  transferred  from  a 
communication  facility  and  converting  it  to  human-readable  form.  Many  REMOTE  BATCH  DEVICEs  have  the  ability 
to  store  input  and/or  output;  some  REMOTE  BATCH  DEVICEs  can  be  employed  as  interactive  consumer  terminals  as 
well.  Printers,  card  readers,  paper  tape  reader/punches,  etc.,  are  frequently  included  as  parts  of  REMOTE  BAT(>H 
DEVICES. 
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REMOTE  BATCH  SERVICE:  A  data  processing  SERVICE  which  can  be  used  by  CONSUMERS,  remote  from  the 
computer,  in  order  to  have  their  submissions  submitted  for  BATCH  PROCESSING.  A  REMOTE  BATCH  SERVICE  is 
sometimes  called  an  RJE  (remote  job  entry)  system  or  a  remote  computer  service.  The  computers  supporting  REMOTE 
BATCH  SERVICES  frequently  also  provide  INTERACTIVE  services  to  remote  consumers. 

REMOTE  SITE:  The  facility  maintained  by  either  the  PROVIDER  or  the  CONSUMER  of  the  REMOTE  BATCH 
SERVICE  to  enable  the  CONSUMER  to  input  submissions  to  the  SERVICE.  The  remote  site  may  be  located  distant 
from  the  CENTRAL  SITE,  or  it  may  be  located  in  the  same  vicinity  as  the  CENTRAL  SITE.  The  remote  site  consists 
of,  at  least,  one  or  more  REMOTE  BATCH  DEVICEs  and  a  means  of  communication  with  the  CENTRAL  SITE.  It  may 
also  include  personnel  to  operate  the  REMOTE  BATCH  DEVICEs. 

REQUEST  OUTPUT  MODE:  In  REQUEST  OUTPUT  MODE  the  CONSUMER  requests  output  for  a  particular  JOB  and 
the  CENTRAL  SITE  responds  by  transmitting  the  requested  output  to  the  designated  REMOTE  BATCH  DEVICE. 

SERVICE:  The  ability  to  meet  the  functional  needs  of  CONSUMERS  of  data  processing.  SERVICE  includes  the  attributes 
of  timeliness,  reliability,  availability,  cost,  and  accuracy.  When  a  data  processing  SERVICE  is  provided  to 
CONSUMERS,  they  need  not  be  concerned  with  the  techniques  for  processing,  the  hardware,  the  computer  languages, 
or  the  identity  of  support  personnel.  Frequently,  the  SERVICE  provided  includes  maintenance  of  programs  and  control 
statements  as  well  as  updating  of  databases;  in  some  cases  it  includes  review  of  the  accuracy  of  reports  and  the  control 
of  financial  data. 

TURNAROUND  TIME:  The  elapsed  time  from  the  beginning  of  INPUT  to  a  REMOTE  BATCH  SERVICE  until  the 
results  of  the  processing  are  available  to  the  CONSUMER. 
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Appendix 

Providers  and  Consumers  of  Remote  Batch  Computer  Service 


A.l  The  Consumers’  Problem 

The  physical  distance  of  the  consumers  from  the  central  site  creates  a  new  dimension  of  non-technical  problems.  In  a 
conventional  batch  service  environment,  consumers  knew  data  processing  personnel  because  they  belonged  to  the  same 
organization — and  probably  worked  in  the  same  building. 

Remote  batch  service,  on  the  other  hand,  may  be  impersonal.  Consumers  frequently  have  no  way  to  determine  the  status 
of  the  system.  If  the  system  is  excessively  sluggish,  or  is  totally  inoperable,  the  consumers  do  not  know  which  of  the  many 
parts  of  the  system  are  at  fault.  They  probably  don’t  need  such  information,  but  being  able  to  obtain  it  may  alleviate 
anxieties  or  make  it  possible  to  expedite  repair.  Unless  special  arrangements  have  been  made,  they  do  not  know  whom  to 
call  to  resolve  their  problems.  And  they  cannot  even  walk  down  the  hall  to  the  data  center  to  find  out  what’s  wrong. 

The  issues  are  too  important  for  responsible  consumers  simply  to  trust  to  fate.  The  technology  is  too  complex  for  them  to 
attempt  to  become  knowledgeable  in  computing  while  dealing  with  their  functional  problems.  The  consumers  are,  therefore, 
faced  with  a  problem:  how  can  they  ensure  acceptable  service  from  the  remote  batch  service  without  trying  to  take  on  all  the 
difficulty  of  learning  enough  to  personally  run  the  data  processing  center?  The  approach  presented  in  this  document  is  to 
identify  the  types  of  metrics  relevant  to  the  consumer’s  viewpoint.  Consumers  can  and  should  use  these  metrics  in  describing 
the  service  quality  they  want  to  receive.  Providers  can  and  should  use  these  metrics  to  describe  the  service  objectives 
provided  to  consumers.  When  a  contractual  relationship  exists  between  provider  and  consumer,  these  metrics  can  and  should 
be  specified  in  that  contract. 

As  stated  in  the  introduction,  the  purpose  of  this  document  is  to  aid  parties,  dealing  in  good  faith,  to  reach  a  common 
agreement  about  the  service  that  will  be  provided  to  remote  consumers,  and  to  determine  whether  the  promised  service  has 
been  provided.  Since  remote  batch  services  are  frequently  incorrectly  characterized,  this  document  includes  definitions  and 
statements  of  limitations.  In  addition,  it  provides  examples  and  explanations  in  order  to  be  explicit  about  implications  of  the 
document.  If  you  are  either  a  consumer  or  provider  of  remote  batch  services,  the  guidelines  provided  here  can  aid  you  in 
negotiations,  selections,  and  evaluations  of  remote  batch  services. 

A. 2  Objectives  of  Provider  vs  Consumer 

A  central  site  is  basically  a  production  facility  that  attempts  to  run  reliably,  with  high  efficiency,  and  at  low  cost.  The 
manager  of  the  central  site  must  be  concerned  with  the  utilization  of  equipment,  its  maintenance,  its  reliability,  and  its  operation. 
Consumers,  on  the  other  hand,  are  concerned  with  what  happened  to  their  particular  submissions.  They  are  annoyed  when  their 
needs  are  not  met  in  a  timely,  accurate  manner  due  to  decisions  by  the  provider — even  when  the  motivation  is  cost 
reduction. 

The  difference  in  concern  and  objective  is  not  the  result  of  a  lack  of  interest  in  the  the  problems,  but  is  due  to  the  variety 
of  objectives  adopted  within  large  organizations.  The  individuals  involved  in  disputes  about  machine  productivity  as  opposed 
to  responsiveness  are  reflecting  the  conflicts  between  these  objectives.  Eoen  when  no  dramatic  conflict  exists,  the  individuals 
involved  in  a  remote  batch  service  may  become  frustrated  in  describing  the  service's  performance  because  they  do  not  know  which 
parts  of  it  to  measure. 

Many  measures  have  been  applied  to  remote  batch  service.  Several  have  already  been  mentioned  in  this  publication.  In 
order  to  reduce  contention,  the  definitions  of  remote  batch  service  quality  should  he  agreed  upon  and 
published  in  advance  at  any  attempt  to  measure  that  service. 
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NBS  TECHNICAL  PUBLICATIONS 


PERIODICALS 

JOURNAL  OF  RESEARCH — The  Journal  of  Research  of  the 
National  Bureau  of  Standards  reports  NBS  research  and  develop¬ 
ment  in  those  disciplines  of  the  physical  and  engineering  sciences  in 
which  the  Bureau  is  active.  These  include  physics,  chemistry, 
engineering,  mathematics,  and  computer  sciences.  Papers  cover  a 
broad  range  of  subjects,  with  major  emphasis  on  measurement 
methodology  and  the  basic  technology  underlying  standardization. 
Also  included  from  time  to  time  are  survey  articles  on  topics 
closely  related  to  the  Bureau’s  technical  and  scientific  programs. 
As  a  special  service  to  subscribers  each  issue  contains  complete 
citations  to  all  recent  Bureau  publications  in  both  NBS  and  non- 
NBS  media.  Issued  six  times  a  year.  Annual  subscription:  domestic 
Sl.T  foreign  516.2.“'.  .Single  copy,  S.J  domestic;  $.T75  foreign. 
NOTE:  The  Journal  was  formerly  published  in  two  sections:  Sec¬ 
tion  A  “Physics  and  Chemistry"  and  Section  B  “Mathematical 
Sciences” 

DIMENSIONS/NBS — This  monthly  magazine  is  published  to  in¬ 
form  scientists,  engineers,  business  and  industry  leaders,  teachers, 
students,  and  consumers  of  the  latest  advances  in  science  and 
technology,  with  primary  emphasis  on  v^ork  at  N  BS.  The  magazine 
highlights  and  reviews  such  issues  as  energy  research,  fire  protec¬ 
tion,  building  technology,  metric  conversion,  pollution  abatement, 
health  and  safety,  and  consumer  product  performance.  In  addi¬ 
tion,  it  reports  the  results  of  Bureau  programs  in  measurement 
standards  and  techniques,  properties  of  matter  and  materials, 
engineering  standards  and  services,  instrumentation,  and 
automatic  data  processing  Annual  subscription,  domestic  $11; 
foreign  $13.75. 

NONPERIODICALS 

Monographs — Major  contributions  to  the  technical  literature  on 
various  subjects  related  to  the  Bureau’s  scientific  and  technical  ac¬ 
tivities. 

Handbooks— Recommended  codes  of  engineering  and  industrial 
practice  (including  safety  codes)  developed  in  cooperation  with  in¬ 
terested  industries,  professional  organizations,  and  regulatory 
bodies. 

Special  Publications — Include  proceedings  of  conferences  spon¬ 
sored  by  NBS,  NBS  annual  reports,  and  other  special  publications 
appropriate  to  this  grouping  such  as  wall  charts,  pocket  cards,  and 
bibliographies. 

Applied  Mathematics  Series — Mathematical  tables,  manuals,  and 
studies  of  special  interest  to  physicists,  engineers,  chemists, 
biologists,  mathematicians,  computer  programmers,  and  others 
engaged  in  scientific  and  technical  work. 

National  Standard  Reference  Data  Series — Provides  quantitative 
data  on  the  physical  and  chemical  properties  of  materials,  com¬ 
piled  from  the  world’s  literature  and  critically  evaluated. 
Developed  under  a  worldwide  program  coordinated  by  NBS  under 
the  authority  of  the  National  Standard  Data  Act  (Public  Law 
90-396). 


NOTE;  The  principal  publication  outlet  for  the  foregoing  data  is 
the  Journal  of  Physical  and  Chemical  Reference  Data  (JPCRD) 
published  quarterly  for  NBS  by  the  American  Chemical  Society 
(ACS)  and  the  American  Institute  of  Physics  (A IP).  Subscriptions, 
reprints,  and  supplements  available  from  ACS,  1 155  Sixteenth  St., 
NW,  Washington,  DC  20056. 

Building  Science  Series — Disseminates  technical  information 
develofied  at  the  Bureau  on  building  materials,  components, 
systems,  and  whole  structures.  The  series  presents  research  results, 
test  methods,  and  performance  criteria  related  to  the  structural  and 
environmental  functions  and  the  durability  and  safety  charac¬ 
teristics  of  building  elements  and  systems. 

Technical  Notes — Studies  or  reports  which  are  complete  in  them¬ 
selves  but  restrictive  in  their  treatment  of  a  subject.  Analogous  to 
monographs  but  not  so  comprehensive  in  scope  or  definitive  in 
treatment  of  the  subject  area.  Often  serve  as  a  vehicle  for  final 
reports  of  work  performed  at  NBS  under  the  sponsorship  of  other 
government  agencies. 

Voluntary  Product  Standards — Developed  under  procedures 
published  by  the  Department  of  Commerce  in  Part  10,  Title  15,  of 
the  Code  of  Federal  Regulations.  The  standards  establish 
nationally  recognized  requirements  for  products,  and  provide  all 
concerned  interests  with  a  basis  for  common  understanding  of  the 
characteristics  of  the  products.  NBS  administers  this  program  as  a 
supplement  to  the  activities  of  the  private  sector  standardizing 
organizations. 

Consumer  Information  Series — Practical  information,  based  on 
N  BS  research  and  experience,  covering  areas  of  interest  to  the  con¬ 
sumer.  Easily  understandable  language  and  illustrations  provide 
useful  background  knowledge  for  shopping  in  today’s  tech¬ 
nological  marketplace. 

Order  the  above  NBS  publications  from:  Superintendent  of  Docu¬ 
ments.  Government  Printing  Office,  Washington,  DC  20402. 

Order  the  following  NBS  publications — FIPS  and  NBS  IP's — from 
the  National  Technical  Information  Services,  Springfield,  VA  22161 

Federal  Information  Processing  Standards  Publications  (FIPS 

PUB)— Publications  in  this  series  collectively  constitute  the 
Federal  Information  Processing  Standards  Register.  The  Register 
serves  as  the  official  source  of  information  in  the  Federal  Govern¬ 
ment  regarding  standards  issued  by  NBS  pursuant  to  the  Federal 
Property  and  Administrative  Services  Act  of  1949  as  amended. 
Public  Law  89-306  (79  Stat.  1127),  and  as  implemented  by  Ex¬ 
ecutive  Order  1 1717  (38  FR  12315,  dated  May  1 1,  1973)  and  Part  6 
of  Title  15  CFR  (Code  of  Federal  Regulations). 

NBS  Interagency  Reports  (NBSIR) — A  special  series  of  interim  or 
final  reports  on  work  performed  by  NBS  for  outside  sponsors 
(both  government  and  non-government).  In  general,  initial  dis¬ 
tribution  is  handled  by  the  sponsor;  public  distribution  is  by  the 
National  Technical  Information  Services,  Springfield,  VA  22161, 
in  paper  copy  or  microfiche  form. 


BIBLIOGRAPHIC  SUBSCRIPTION  SERVICES 


The  following  current-awareness  and  literature-survey  bibliographies 
are  issued  periodically  by  the  Bureau: 

Cryogenic  Data  Center  Current  Awareness  Service.  A  literature  sur¬ 
vey  issued  biweekly.  Annual  subscription:  domestic  $35;  foreign 
$45. 

Liquefied  Natural  (ias.  A  literature  survey  issued  quarterly.  Annual 
subscription:  $30. 


Superconducting  Devices  and  Materials.  A  literature  survey  issued 
quarterly.  .Annual  subscription:  $45.  Please  send  subscription  or¬ 
ders  and  remittances  for  the  preceding  bibliographic  services  to  the 
National  Bureau  of  Standards,  Cryogenic  Data  Center  (736) 
Boulder,  CO  80303. 
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